2024-09-12T06:15:54.919 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-12T06:15:54.939 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-12T06:15:55.036 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/dis-2024-09-12_04:46:10-rbd-wip-dis-testing-distro-default-smithi/7902364 branch: wip-dis-testing description: rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{centos_latest} workloads/rbd-mirror-stress-workunit-min-compat-client-octopus} email: idryomov@gmail.com first_in_suite: false job_id: '7902364' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi meta: - desc: run two ceph clusters and install rbd-mirror - desc: configure the permissions for client.mirror - desc: configure the permissions for client.mirror - desc: 2 ceph clusters with 1 mon and 3 osds each - desc: run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon name: dis-2024-09-12_04:46:10-rbd-wip-dis-testing-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 30 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-dis-testing ceph: conf: client: debug rbd: 20 debug rbd_mirror: 15 log to stderr: false rbd default features: 125 client.mirror.0: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.1: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.2: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.3: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.4: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.5: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.6: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid global: mon client directed command retry: 5 mon warn on pool no app: false ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bluestore block size: 96636764160 bluestore compression algorithm: lz4 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME sha1: d61b08865d4d98738f45ff2e4806d7fb6d03eca1 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: d61b08865d4d98738f45ff2e4806d7fb6d03eca1 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-dis-testing sha1: d61b08865d4d98738f45ff2e4806d7fb6d03eca1 owner: scheduled_dis@teuthology priority: 70 repo: https://github.com/ceph/ceph-ci roles: - - cluster1.mon.a - cluster1.mgr.x - cluster2.mgr.x - cluster1.osd.0 - cluster1.osd.1 - cluster1.osd.2 - cluster1.client.0 - cluster2.client.0 - - cluster2.mon.a - cluster2.osd.0 - cluster2.osd.1 - cluster2.osd.2 - cluster1.client.mirror - cluster1.client.mirror.0 - cluster1.client.mirror.1 - cluster1.client.mirror.2 - cluster1.client.mirror.3 - cluster1.client.mirror.4 - cluster1.client.mirror.5 - cluster1.client.mirror.6 - cluster2.client.mirror - cluster2.client.mirror.0 - cluster2.client.mirror.1 - cluster2.client.mirror.2 - cluster2.client.mirror.3 - cluster2.client.mirror.4 - cluster2.client.mirror.5 - cluster2.client.mirror.6 seed: 1680 sha1: d61b08865d4d98738f45ff2e4806d7fb6d03eca1 sleep_before_teardown: 0 subset: 42/100 suite: rbd suite_branch: wip-dis-testing suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_d61b08865d4d98738f45ff2e4806d7fb6d03eca1/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci suite_sha1: d61b08865d4d98738f45ff2e4806d7fb6d03eca1 targets: smithi096.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF5PABoXj0XD5QU3fRJj4OmIZGRIOMCS4DbBo8Dsdz7RZ6jxRAn4d86LXkqXIH+TJbhbJ5M+gn089Cqb92T+yFI= smithi152.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBH6VISxqa8TYG4EzTv7+6lZ5JEAW0zEQsltSRY1l4Q2cK9o7oz4wS80C5VOQs74JCgsa8UBd2xTcMSPKgaUvUvI= tasks: - install: extra_packages: - rbd-mirror - ceph: cluster: cluster1 - ceph: cluster: cluster2 - exec: cluster1.client.mirror.0: - sudo ceph --cluster cluster1 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd' - exec: cluster1.client.mirror.0: - sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - workunit: clients: cluster1.client.mirror: - rbd/rbd_mirror_stress.sh env: CEPH_ARGS: '' RBD_MIRROR_MIN_COMPAT_CLIENT: octopus RBD_MIRROR_USE_EXISTING_CLUSTER: '1' teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-12_04:46:10 tube: smithi user: dis verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-12T06:15:55.037 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_d61b08865d4d98738f45ff2e4806d7fb6d03eca1/qa; will attempt to use it 2024-09-12T06:15:55.038 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_d61b08865d4d98738f45ff2e4806d7fb6d03eca1/qa/tasks 2024-09-12T06:15:55.038 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-12T06:15:55.039 INFO:teuthology.task.internal:Checking packages... 2024-09-12T06:15:55.058 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'd61b08865d4d98738f45ff2e4806d7fb6d03eca1' 2024-09-12T06:15:55.059 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-12T06:15:55.059 INFO:teuthology.packaging:ref: None 2024-09-12T06:15:55.059 INFO:teuthology.packaging:tag: None 2024-09-12T06:15:55.059 INFO:teuthology.packaging:branch: wip-dis-testing 2024-09-12T06:15:55.059 INFO:teuthology.packaging:sha1: d61b08865d4d98738f45ff2e4806d7fb6d03eca1 2024-09-12T06:15:55.059 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-dis-testing 2024-09-12T06:15:55.232 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4883.gd61b0886 2024-09-12T06:15:55.234 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-12T06:15:55.241 INFO:teuthology.task.internal:no buildpackages task found 2024-09-12T06:15:55.241 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-12T06:15:55.257 INFO:teuthology.task.internal:Saving configuration 2024-09-12T06:15:55.272 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-12T06:15:55.280 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-12T06:15:55.314 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi096.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dis-2024-09-12_04:46:10-rbd-wip-dis-testing-distro-default-smithi/7902364', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-12 06:10:35.028511', 'locked_by': 'scheduled_dis@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF5PABoXj0XD5QU3fRJj4OmIZGRIOMCS4DbBo8Dsdz7RZ6jxRAn4d86LXkqXIH+TJbhbJ5M+gn089Cqb92T+yFI='} 2024-09-12T06:15:55.346 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi152.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dis-2024-09-12_04:46:10-rbd-wip-dis-testing-distro-default-smithi/7902364', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-12 06:10:35.030112', 'locked_by': 'scheduled_dis@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBH6VISxqa8TYG4EzTv7+6lZ5JEAW0zEQsltSRY1l4Q2cK9o7oz4wS80C5VOQs74JCgsa8UBd2xTcMSPKgaUvUvI='} 2024-09-12T06:15:55.347 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-12T06:15:55.353 INFO:teuthology.task.internal:roles: ubuntu@smithi096.front.sepia.ceph.com - ['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'] 2024-09-12T06:15:55.353 INFO:teuthology.task.internal:roles: ubuntu@smithi152.front.sepia.ceph.com - ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6'] 2024-09-12T06:15:55.353 INFO:teuthology.run_tasks:Running task console_log... 2024-09-12T06:15:55.441 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f00f8b5fac0>, signals=[15]) 2024-09-12T06:15:55.442 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-12T06:15:55.452 INFO:teuthology.task.internal:Opening connections... 2024-09-12T06:15:55.452 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi096.front.sepia.ceph.com 2024-09-12T06:15:55.453 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi096.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-12T06:15:55.538 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi152.front.sepia.ceph.com 2024-09-12T06:15:55.538 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-12T06:15:55.621 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-12T06:15:55.626 DEBUG:teuthology.orchestra.run.smithi096:> uname -m 2024-09-12T06:15:55.644 INFO:teuthology.orchestra.run.smithi096.stdout:x86_64 2024-09-12T06:15:55.644 DEBUG:teuthology.orchestra.run.smithi096:> cat /etc/os-release 2024-09-12T06:15:55.702 INFO:teuthology.orchestra.run.smithi096.stdout:NAME="CentOS Stream" 2024-09-12T06:15:55.702 INFO:teuthology.orchestra.run.smithi096.stdout:VERSION="9" 2024-09-12T06:15:55.702 INFO:teuthology.orchestra.run.smithi096.stdout:ID="centos" 2024-09-12T06:15:55.702 INFO:teuthology.orchestra.run.smithi096.stdout:ID_LIKE="rhel fedora" 2024-09-12T06:15:55.703 INFO:teuthology.orchestra.run.smithi096.stdout:VERSION_ID="9" 2024-09-12T06:15:55.703 INFO:teuthology.orchestra.run.smithi096.stdout:PLATFORM_ID="platform:el9" 2024-09-12T06:15:55.703 INFO:teuthology.orchestra.run.smithi096.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-12T06:15:55.703 INFO:teuthology.orchestra.run.smithi096.stdout:ANSI_COLOR="0;31" 2024-09-12T06:15:55.703 INFO:teuthology.orchestra.run.smithi096.stdout:LOGO="fedora-logo-icon" 2024-09-12T06:15:55.703 INFO:teuthology.orchestra.run.smithi096.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-12T06:15:55.703 INFO:teuthology.orchestra.run.smithi096.stdout:HOME_URL="https://centos.org/" 2024-09-12T06:15:55.703 INFO:teuthology.orchestra.run.smithi096.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-12T06:15:55.703 INFO:teuthology.orchestra.run.smithi096.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-12T06:15:55.703 INFO:teuthology.orchestra.run.smithi096.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-12T06:15:55.704 INFO:teuthology.lock.ops:Updating smithi096.front.sepia.ceph.com on lock server 2024-09-12T06:15:55.737 DEBUG:teuthology.orchestra.run.smithi152:> uname -m 2024-09-12T06:15:55.755 INFO:teuthology.orchestra.run.smithi152.stdout:x86_64 2024-09-12T06:15:55.756 DEBUG:teuthology.orchestra.run.smithi152:> cat /etc/os-release 2024-09-12T06:15:55.813 INFO:teuthology.orchestra.run.smithi152.stdout:NAME="CentOS Stream" 2024-09-12T06:15:55.813 INFO:teuthology.orchestra.run.smithi152.stdout:VERSION="9" 2024-09-12T06:15:55.813 INFO:teuthology.orchestra.run.smithi152.stdout:ID="centos" 2024-09-12T06:15:55.813 INFO:teuthology.orchestra.run.smithi152.stdout:ID_LIKE="rhel fedora" 2024-09-12T06:15:55.814 INFO:teuthology.orchestra.run.smithi152.stdout:VERSION_ID="9" 2024-09-12T06:15:55.814 INFO:teuthology.orchestra.run.smithi152.stdout:PLATFORM_ID="platform:el9" 2024-09-12T06:15:55.814 INFO:teuthology.orchestra.run.smithi152.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-12T06:15:55.814 INFO:teuthology.orchestra.run.smithi152.stdout:ANSI_COLOR="0;31" 2024-09-12T06:15:55.814 INFO:teuthology.orchestra.run.smithi152.stdout:LOGO="fedora-logo-icon" 2024-09-12T06:15:55.814 INFO:teuthology.orchestra.run.smithi152.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-12T06:15:55.814 INFO:teuthology.orchestra.run.smithi152.stdout:HOME_URL="https://centos.org/" 2024-09-12T06:15:55.814 INFO:teuthology.orchestra.run.smithi152.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-12T06:15:55.814 INFO:teuthology.orchestra.run.smithi152.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-12T06:15:55.814 INFO:teuthology.orchestra.run.smithi152.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-12T06:15:55.814 INFO:teuthology.lock.ops:Updating smithi152.front.sepia.ceph.com on lock server 2024-09-12T06:15:55.844 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-12T06:15:55.854 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-12T06:15:55.861 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-12T06:15:55.861 DEBUG:teuthology.orchestra.run.smithi096:> test '!' -e /home/ubuntu/cephtest 2024-09-12T06:15:55.864 DEBUG:teuthology.orchestra.run.smithi152:> test '!' -e /home/ubuntu/cephtest 2024-09-12T06:15:55.881 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-12T06:15:55.888 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-12T06:15:55.888 DEBUG:teuthology.orchestra.run.smithi096:> test -z $(ls -A /var/lib/ceph) 2024-09-12T06:15:55.921 DEBUG:teuthology.orchestra.run.smithi152:> test -z $(ls -A /var/lib/ceph) 2024-09-12T06:15:56.048 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-12T06:15:56.116 INFO:teuthology.run_tasks:Running task kernel... 2024-09-12T06:15:56.131 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-12T06:15:56.131 INFO:teuthology.task.kernel:config {'cluster1.mon.a': {'kdb': 1, 'sha1': 'distro'}, 'cluster2.mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-12T06:15:56.132 DEBUG:teuthology.orchestra.run.smithi096:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-12T06:15:56.132 DEBUG:teuthology.orchestra.run.smithi152:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-12T06:15:56.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-12T06:15:56.150 DEBUG:teuthology.orchestra.run.smithi096:> uname -r 2024-09-12T06:15:56.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-12T06:15:56.151 DEBUG:teuthology.orchestra.run.smithi152:> uname -r 2024-09-12T06:15:56.207 INFO:teuthology.orchestra.run.smithi096.stdout:5.14.0-503.el9.x86_64 2024-09-12T06:15:56.207 INFO:teuthology.task.kernel:Running kernel on smithi096: 5.14.0-503.el9.x86_64 2024-09-12T06:15:56.207 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum install -y kernel 2024-09-12T06:15:56.208 INFO:teuthology.orchestra.run.smithi152.stdout:5.14.0-503.el9.x86_64 2024-09-12T06:15:56.208 INFO:teuthology.task.kernel:Running kernel on smithi152: 5.14.0-503.el9.x86_64 2024-09-12T06:15:56.209 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum install -y kernel 2024-09-12T06:15:58.715 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.2 MB 00:00 2024-09-12T06:15:58.919 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.2 MB 00:00 2024-09-12T06:16:03.757 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - AppStream 5.2 MB/s | 20 MB 00:03 2024-09-12T06:16:22.196 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - CRB 455 kB/s | 6.5 MB 00:14 2024-09-12T06:16:24.554 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - Extras packages 16 kB/s | 19 kB 00:01 2024-09-12T06:16:25.489 INFO:teuthology.orchestra.run.smithi096.stdout:Extra Packages for Enterprise Linux 28 MB/s | 23 MB 00:00 2024-09-12T06:16:30.939 INFO:teuthology.orchestra.run.smithi096.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-12T06:16:32.724 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - AppStream 639 kB/s | 20 MB 00:32 2024-09-12T06:16:32.920 INFO:teuthology.orchestra.run.smithi096.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-12T06:16:32.920 INFO:teuthology.orchestra.run.smithi096.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-12T06:16:33.000 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-12T06:16:33.011 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-12T06:16:33.011 INFO:teuthology.orchestra.run.smithi096.stdout: Package Architecture Version Repository Size 2024-09-12T06:16:33.011 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-12T06:16:33.011 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-09-12T06:16:33.011 INFO:teuthology.orchestra.run.smithi096.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-12T06:16:33.011 INFO:teuthology.orchestra.run.smithi096.stdout:Installing dependencies: 2024-09-12T06:16:33.011 INFO:teuthology.orchestra.run.smithi096.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-12T06:16:33.011 INFO:teuthology.orchestra.run.smithi096.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-12T06:16:33.011 INFO:teuthology.orchestra.run.smithi096.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-12T06:16:33.011 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:16:33.011 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-09-12T06:16:33.012 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-12T06:16:33.012 INFO:teuthology.orchestra.run.smithi096.stdout:Install 4 Packages 2024-09-12T06:16:33.012 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:16:33.012 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 79 M 2024-09-12T06:16:33.012 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 126 M 2024-09-12T06:16:33.012 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-09-12T06:16:33.232 INFO:teuthology.orchestra.run.smithi096.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 187 kB/s | 23 kB 00:00 2024-09-12T06:16:38.333 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - CRB 3.7 MB/s | 6.5 MB 00:01 2024-09-12T06:16:39.975 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - Extras packages 42 kB/s | 19 kB 00:00 2024-09-12T06:16:40.281 INFO:teuthology.orchestra.run.smithi096.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 2.2 MB/s | 16 MB 00:07 2024-09-12T06:16:41.119 INFO:teuthology.orchestra.run.smithi152.stdout:Extra Packages for Enterprise Linux 22 MB/s | 23 MB 00:01 2024-09-12T06:16:46.055 INFO:teuthology.orchestra.run.smithi096.stdout:(3/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 2.7 MB/s | 35 MB 00:12 2024-09-12T06:16:46.397 INFO:teuthology.orchestra.run.smithi096.stdout:(4/4): kernel-modules-core-5.14.0-505.el9.x86_6 2.2 MB/s | 29 MB 00:13 2024-09-12T06:16:46.397 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-09-12T06:16:46.397 INFO:teuthology.orchestra.run.smithi096.stdout:Total 5.9 MB/s | 79 MB 00:13 2024-09-12T06:16:46.435 INFO:teuthology.orchestra.run.smithi152.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-12T06:16:47.080 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-09-12T06:16:47.427 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-09-12T06:16:47.427 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-09-12T06:16:48.386 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-09-12T06:16:48.387 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-09-12T06:16:48.442 INFO:teuthology.orchestra.run.smithi152.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-12T06:16:48.442 INFO:teuthology.orchestra.run.smithi152.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-12T06:16:48.539 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-09-12T06:16:48.545 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-12T06:16:48.545 INFO:teuthology.orchestra.run.smithi152.stdout: Package Architecture Version Repository Size 2024-09-12T06:16:48.545 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-12T06:16:48.545 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-09-12T06:16:48.545 INFO:teuthology.orchestra.run.smithi152.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-12T06:16:48.545 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-09-12T06:16:48.546 INFO:teuthology.orchestra.run.smithi152.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-12T06:16:48.546 INFO:teuthology.orchestra.run.smithi152.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-12T06:16:48.546 INFO:teuthology.orchestra.run.smithi152.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-12T06:16:48.546 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:16:48.546 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-09-12T06:16:48.546 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-12T06:16:48.546 INFO:teuthology.orchestra.run.smithi152.stdout:Install 4 Packages 2024-09-12T06:16:48.546 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:16:48.547 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 79 M 2024-09-12T06:16:48.547 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 126 M 2024-09-12T06:16:48.547 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-09-12T06:16:48.895 INFO:teuthology.orchestra.run.smithi152.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 320 kB/s | 23 kB 00:00 2024-09-12T06:16:50.848 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-09-12T06:16:51.549 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-12T06:16:51.632 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-12T06:16:52.745 INFO:teuthology.orchestra.run.smithi152.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 4.0 MB/s | 16 MB 00:03 2024-09-12T06:16:53.295 INFO:teuthology.orchestra.run.smithi152.stdout:(3/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 7.7 MB/s | 35 MB 00:04 2024-09-12T06:16:53.587 INFO:teuthology.orchestra.run.smithi152.stdout:(4/4): kernel-modules-core-5.14.0-505.el9.x86_6 6.1 MB/s | 29 MB 00:04 2024-09-12T06:16:53.587 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-09-12T06:16:53.587 INFO:teuthology.orchestra.run.smithi152.stdout:Total 16 MB/s | 79 MB 00:05 2024-09-12T06:16:54.268 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-09-12T06:16:54.436 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-12T06:16:54.567 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-12T06:16:54.629 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-09-12T06:16:54.629 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-09-12T06:16:55.626 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-09-12T06:16:55.627 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-09-12T06:16:58.090 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-09-12T06:16:58.764 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-12T06:16:58.864 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-12T06:16:59.366 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-12T06:16:59.399 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-12T06:17:01.654 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-12T06:17:01.785 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-12T06:17:04.229 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-12T06:17:06.603 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-12T06:17:06.835 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-12T06:17:11.658 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-12T06:17:42.627 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-12T06:17:42.668 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-12T06:17:43.049 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-12T06:17:43.049 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-12T06:17:43.049 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-12T06:17:43.049 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-12T06:17:43.918 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-12T06:17:43.918 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:17:43.918 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-09-12T06:17:43.918 INFO:teuthology.orchestra.run.smithi096.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-12T06:17:43.918 INFO:teuthology.orchestra.run.smithi096.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-12T06:17:43.918 INFO:teuthology.orchestra.run.smithi096.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-12T06:17:43.918 INFO:teuthology.orchestra.run.smithi096.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-12T06:17:43.919 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:17:43.919 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-12T06:17:44.275 DEBUG:teuthology.orchestra.run.smithi096:> rpm -q kernel | sort -rV | head -n 1 2024-09-12T06:17:44.358 INFO:teuthology.orchestra.run.smithi096.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-12T06:17:44.358 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-12T06:17:44.359 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-12T06:17:44.359 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-12T06:17:44.359 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-12T06:17:44.359 INFO:teuthology.task.kernel:Installing distro kernel on cluster1.mon.a... 2024-09-12T06:17:44.359 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi096.front.sepia.ceph.com, path=None, version=distro) 2024-09-12T06:17:44.359 DEBUG:teuthology.orchestra.run.smithi096:> rpm -q kernel | sort -rV | head -n 1 2024-09-12T06:17:44.418 INFO:teuthology.orchestra.run.smithi096.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-12T06:17:44.418 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-12T06:17:44.418 DEBUG:teuthology.orchestra.run.smithi096:> sudo rpm -qi grub2-tools 2024-09-12T06:17:44.455 INFO:teuthology.orchestra.run.smithi096.stdout:Name : grub2-tools 2024-09-12T06:17:44.455 INFO:teuthology.orchestra.run.smithi096.stdout:Epoch : 1 2024-09-12T06:17:44.455 INFO:teuthology.orchestra.run.smithi096.stdout:Version : 2.06 2024-09-12T06:17:44.456 INFO:teuthology.orchestra.run.smithi096.stdout:Release : 82.el9 2024-09-12T06:17:44.456 INFO:teuthology.orchestra.run.smithi096.stdout:Architecture: x86_64 2024-09-12T06:17:44.456 INFO:teuthology.orchestra.run.smithi096.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-12T06:17:44.456 INFO:teuthology.orchestra.run.smithi096.stdout:Group : Unspecified 2024-09-12T06:17:44.456 INFO:teuthology.orchestra.run.smithi096.stdout:Size : 8274670 2024-09-12T06:17:44.456 INFO:teuthology.orchestra.run.smithi096.stdout:License : GPLv3+ 2024-09-12T06:17:44.456 INFO:teuthology.orchestra.run.smithi096.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-12T06:17:44.456 INFO:teuthology.orchestra.run.smithi096.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-12T06:17:44.456 INFO:teuthology.orchestra.run.smithi096.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-12T06:17:44.456 INFO:teuthology.orchestra.run.smithi096.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-12T06:17:44.456 INFO:teuthology.orchestra.run.smithi096.stdout:Packager : builder@centos.org 2024-09-12T06:17:44.456 INFO:teuthology.orchestra.run.smithi096.stdout:Vendor : CentOS 2024-09-12T06:17:44.456 INFO:teuthology.orchestra.run.smithi096.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-12T06:17:44.457 INFO:teuthology.orchestra.run.smithi096.stdout:Summary : Support tools for GRUB. 2024-09-12T06:17:44.457 INFO:teuthology.orchestra.run.smithi096.stdout:Description : 2024-09-12T06:17:44.457 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:17:44.457 INFO:teuthology.orchestra.run.smithi096.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-12T06:17:44.457 INFO:teuthology.orchestra.run.smithi096.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-12T06:17:44.457 INFO:teuthology.orchestra.run.smithi096.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-12T06:17:44.457 INFO:teuthology.orchestra.run.smithi096.stdout:hardware devices. 2024-09-12T06:17:44.457 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:17:44.457 INFO:teuthology.orchestra.run.smithi096.stdout:This subpackage provides tools for support of all platforms. 2024-09-12T06:17:44.458 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-12T06:17:44.458 INFO:teuthology.task.kernel:Updating grub on smithi096 to boot 5.14.0-505.el9.x86_64 2024-09-12T06:17:44.458 DEBUG:teuthology.orchestra.run.smithi096:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-12T06:17:45.089 INFO:teuthology.orchestra.run.smithi096.stderr:Generating grub configuration file ... 2024-09-12T06:17:46.447 INFO:teuthology.orchestra.run.smithi096.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-12T06:17:46.490 INFO:teuthology.orchestra.run.smithi096.stderr:done 2024-09-12T06:17:46.493 DEBUG:teuthology.orchestra.run.smithi096:> mktemp 2024-09-12T06:17:46.508 INFO:teuthology.orchestra.run.smithi096.stdout:/tmp/tmp.EslhblIWnE 2024-09-12T06:17:46.508 DEBUG:teuthology.orchestra.run.smithi096:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.EslhblIWnE 2024-09-12T06:17:46.584 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod 0666 /tmp/tmp.EslhblIWnE 2024-09-12T06:17:46.733 DEBUG:teuthology.orchestra.remote:smithi096:/tmp/tmp.EslhblIWnE is 6KB 2024-09-12T06:17:46.753 DEBUG:teuthology.orchestra.run.smithi096:> rm -fr /tmp/tmp.EslhblIWnE 2024-09-12T06:17:46.768 DEBUG:teuthology.orchestra.run.smithi096:> sudo /bin/ls /boot/loader/entries || true 2024-09-12T06:17:46.831 INFO:teuthology.orchestra.run.smithi096.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-12T06:17:46.878 INFO:teuthology.orchestra.run.smithi096.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-12T06:17:46.878 INFO:teuthology.orchestra.run.smithi096.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-12T06:17:46.878 INFO:teuthology.orchestra.run.smithi096.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-12T06:17:46.880 DEBUG:teuthology.orchestra.run.smithi096:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-12T06:17:47.030 DEBUG:teuthology.orchestra.run.smithi096:> sudo shutdown -r now 2024-09-12T06:17:49.592 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-12T06:17:49.632 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-12T06:17:49.901 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-12T06:17:49.901 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-12T06:17:49.901 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-12T06:17:49.901 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-12T06:17:50.712 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-12T06:17:50.712 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:17:50.712 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-09-12T06:17:50.712 INFO:teuthology.orchestra.run.smithi152.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-12T06:17:50.712 INFO:teuthology.orchestra.run.smithi152.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-12T06:17:50.712 INFO:teuthology.orchestra.run.smithi152.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-12T06:17:50.712 INFO:teuthology.orchestra.run.smithi152.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-12T06:17:50.712 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:17:50.713 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-09-12T06:17:51.053 DEBUG:teuthology.orchestra.run.smithi152:> rpm -q kernel | sort -rV | head -n 1 2024-09-12T06:17:51.135 INFO:teuthology.orchestra.run.smithi152.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-12T06:17:51.135 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-12T06:17:51.136 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-12T06:17:51.136 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-12T06:17:51.136 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-12T06:17:51.136 INFO:teuthology.task.kernel:Installing distro kernel on cluster2.mon.a... 2024-09-12T06:17:51.136 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi152.front.sepia.ceph.com, path=None, version=distro) 2024-09-12T06:17:51.136 DEBUG:teuthology.orchestra.run.smithi152:> rpm -q kernel | sort -rV | head -n 1 2024-09-12T06:17:51.195 INFO:teuthology.orchestra.run.smithi152.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-12T06:17:51.195 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-12T06:17:51.196 DEBUG:teuthology.orchestra.run.smithi152:> sudo rpm -qi grub2-tools 2024-09-12T06:17:51.227 INFO:teuthology.orchestra.run.smithi152.stdout:Name : grub2-tools 2024-09-12T06:17:51.227 INFO:teuthology.orchestra.run.smithi152.stdout:Epoch : 1 2024-09-12T06:17:51.227 INFO:teuthology.orchestra.run.smithi152.stdout:Version : 2.06 2024-09-12T06:17:51.227 INFO:teuthology.orchestra.run.smithi152.stdout:Release : 82.el9 2024-09-12T06:17:51.228 INFO:teuthology.orchestra.run.smithi152.stdout:Architecture: x86_64 2024-09-12T06:17:51.228 INFO:teuthology.orchestra.run.smithi152.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-12T06:17:51.228 INFO:teuthology.orchestra.run.smithi152.stdout:Group : Unspecified 2024-09-12T06:17:51.228 INFO:teuthology.orchestra.run.smithi152.stdout:Size : 8274670 2024-09-12T06:17:51.228 INFO:teuthology.orchestra.run.smithi152.stdout:License : GPLv3+ 2024-09-12T06:17:51.228 INFO:teuthology.orchestra.run.smithi152.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-12T06:17:51.228 INFO:teuthology.orchestra.run.smithi152.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-12T06:17:51.228 INFO:teuthology.orchestra.run.smithi152.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-12T06:17:51.228 INFO:teuthology.orchestra.run.smithi152.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-12T06:17:51.228 INFO:teuthology.orchestra.run.smithi152.stdout:Packager : builder@centos.org 2024-09-12T06:17:51.228 INFO:teuthology.orchestra.run.smithi152.stdout:Vendor : CentOS 2024-09-12T06:17:51.228 INFO:teuthology.orchestra.run.smithi152.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-12T06:17:51.229 INFO:teuthology.orchestra.run.smithi152.stdout:Summary : Support tools for GRUB. 2024-09-12T06:17:51.229 INFO:teuthology.orchestra.run.smithi152.stdout:Description : 2024-09-12T06:17:51.229 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:17:51.229 INFO:teuthology.orchestra.run.smithi152.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-12T06:17:51.229 INFO:teuthology.orchestra.run.smithi152.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-12T06:17:51.229 INFO:teuthology.orchestra.run.smithi152.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-12T06:17:51.229 INFO:teuthology.orchestra.run.smithi152.stdout:hardware devices. 2024-09-12T06:17:51.229 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:17:51.229 INFO:teuthology.orchestra.run.smithi152.stdout:This subpackage provides tools for support of all platforms. 2024-09-12T06:17:51.230 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-12T06:17:51.230 INFO:teuthology.task.kernel:Updating grub on smithi152 to boot 5.14.0-505.el9.x86_64 2024-09-12T06:17:51.230 DEBUG:teuthology.orchestra.run.smithi152:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-12T06:17:51.976 INFO:teuthology.orchestra.run.smithi152.stderr:Generating grub configuration file ... 2024-09-12T06:17:53.350 INFO:teuthology.orchestra.run.smithi152.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-12T06:17:53.398 INFO:teuthology.orchestra.run.smithi152.stderr:done 2024-09-12T06:17:53.400 DEBUG:teuthology.orchestra.run.smithi152:> mktemp 2024-09-12T06:17:53.416 INFO:teuthology.orchestra.run.smithi152.stdout:/tmp/tmp.wEkfiAJUwA 2024-09-12T06:17:53.416 DEBUG:teuthology.orchestra.run.smithi152:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.wEkfiAJUwA 2024-09-12T06:17:53.483 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod 0666 /tmp/tmp.wEkfiAJUwA 2024-09-12T06:17:53.616 DEBUG:teuthology.orchestra.remote:smithi152:/tmp/tmp.wEkfiAJUwA is 6KB 2024-09-12T06:17:53.637 DEBUG:teuthology.orchestra.run.smithi152:> rm -fr /tmp/tmp.wEkfiAJUwA 2024-09-12T06:17:53.652 DEBUG:teuthology.orchestra.run.smithi152:> sudo /bin/ls /boot/loader/entries || true 2024-09-12T06:17:53.721 INFO:teuthology.orchestra.run.smithi152.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-12T06:17:53.721 INFO:teuthology.orchestra.run.smithi152.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-12T06:17:53.721 INFO:teuthology.orchestra.run.smithi152.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-12T06:17:53.721 INFO:teuthology.orchestra.run.smithi152.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-12T06:17:53.723 DEBUG:teuthology.orchestra.run.smithi152:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-12T06:17:53.880 DEBUG:teuthology.orchestra.run.smithi152:> sudo shutdown -r now 2024-09-12T06:18:17.055 INFO:teuthology.task.kernel:Checking client cluster1.mon.a for new kernel version... 2024-09-12T06:18:17.056 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi096.front.sepia.ceph.com' 2024-09-12T06:18:17.057 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi096.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-12T06:18:23.890 INFO:teuthology.task.kernel:Checking client cluster2.mon.a for new kernel version... 2024-09-12T06:18:23.891 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi152.front.sepia.ceph.com' 2024-09-12T06:18:23.891 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-12T06:18:35.453 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.96 2024-09-12T06:18:42.365 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.152 2024-09-12T06:18:44.457 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi096.front.sepia.ceph.com' 2024-09-12T06:18:44.458 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi096.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-12T06:18:47.521 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.96 2024-09-12T06:18:51.370 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi152.front.sepia.ceph.com' 2024-09-12T06:18:51.371 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-12T06:18:54.429 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.152 2024-09-12T06:18:59.526 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi096.front.sepia.ceph.com' 2024-09-12T06:18:59.527 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi096.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-12T06:19:06.436 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi152.front.sepia.ceph.com' 2024-09-12T06:19:06.437 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-12T06:19:17.949 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.96 2024-09-12T06:19:32.965 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi096.front.sepia.ceph.com' 2024-09-12T06:19:32.966 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi096.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-12T06:19:33.342 DEBUG:teuthology.orchestra.run.smithi096:> true 2024-09-12T06:19:33.723 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi096.front.sepia.ceph.com' 2024-09-12T06:19:33.723 INFO:teuthology.task.kernel:Checking kernel version of cluster1.mon.a, want "5.14.0-505.el9.x86_64"... 2024-09-12T06:19:33.724 DEBUG:teuthology.orchestra.run.smithi096:> uname -r 2024-09-12T06:19:33.738 INFO:teuthology.orchestra.run.smithi096.stdout:5.14.0-505.el9.x86_64 2024-09-12T06:19:33.738 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-12T06:19:33.738 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-12T06:19:33.738 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-12T06:19:34.739 INFO:teuthology.task.kernel:Enabling kdb on cluster1.mon.a... 2024-09-12T06:19:34.740 DEBUG:teuthology.orchestra.run.smithi096:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-12T06:19:34.842 INFO:teuthology.orchestra.run.smithi096.stdout:ttyS1 2024-09-12T06:19:34.864 DEBUG:teuthology.parallel:result is None 2024-09-12T06:20:06.449 DEBUG:teuthology.orchestra.remote:timed out 2024-09-12T06:20:21.449 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi152.front.sepia.ceph.com' 2024-09-12T06:20:21.451 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-12T06:20:21.765 DEBUG:teuthology.orchestra.run.smithi152:> true 2024-09-12T06:20:22.146 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi152.front.sepia.ceph.com' 2024-09-12T06:20:22.147 INFO:teuthology.task.kernel:Checking kernel version of cluster2.mon.a, want "5.14.0-505.el9.x86_64"... 2024-09-12T06:20:22.147 DEBUG:teuthology.orchestra.run.smithi152:> uname -r 2024-09-12T06:20:22.163 INFO:teuthology.orchestra.run.smithi152.stdout:5.14.0-505.el9.x86_64 2024-09-12T06:20:22.163 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-12T06:20:22.163 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-12T06:20:22.163 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-12T06:20:23.164 INFO:teuthology.task.kernel:Enabling kdb on cluster2.mon.a... 2024-09-12T06:20:23.165 DEBUG:teuthology.orchestra.run.smithi152:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-12T06:20:23.242 INFO:teuthology.orchestra.run.smithi152.stdout:ttyS1 2024-09-12T06:20:23.266 DEBUG:teuthology.parallel:result is None 2024-09-12T06:20:23.266 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-12T06:20:23.276 INFO:teuthology.task.internal:Creating test directory... 2024-09-12T06:20:23.276 DEBUG:teuthology.orchestra.run.smithi096:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-12T06:20:23.280 DEBUG:teuthology.orchestra.run.smithi152:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-12T06:20:23.298 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-12T06:20:23.304 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-12T06:20:23.315 INFO:teuthology.task.internal:Creating archive directory... 2024-09-12T06:20:23.315 DEBUG:teuthology.orchestra.run.smithi096:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-12T06:20:23.340 DEBUG:teuthology.orchestra.run.smithi152:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-12T06:20:23.424 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-12T06:20:23.432 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-12T06:20:23.432 DEBUG:teuthology.orchestra.run.smithi096:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-12T06:20:23.435 DEBUG:teuthology.orchestra.run.smithi152:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-12T06:20:23.469 INFO:teuthology.orchestra.run.smithi096.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-12T06:20:23.480 INFO:teuthology.orchestra.run.smithi152.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-12T06:20:23.481 INFO:teuthology.orchestra.run.smithi096.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-12T06:20:23.491 INFO:teuthology.orchestra.run.smithi152.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-12T06:20:23.492 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-12T06:20:23.499 INFO:teuthology.task.internal:Configuring sudo... 2024-09-12T06:20:23.500 DEBUG:teuthology.orchestra.run.smithi096:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-12T06:20:23.526 DEBUG:teuthology.orchestra.run.smithi152:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-12T06:20:23.553 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-12T06:20:23.562 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-12T06:20:23.562 DEBUG:teuthology.orchestra.run.smithi096:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-12T06:20:23.595 DEBUG:teuthology.orchestra.run.smithi152:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-12T06:20:23.611 DEBUG:teuthology.orchestra.run.smithi096:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-12T06:20:23.701 DEBUG:teuthology.orchestra.run.smithi096:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-12T06:20:23.748 DEBUG:teuthology.orchestra.run.smithi096:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-12T06:20:23.841 DEBUG:teuthology.orchestra.run.smithi096:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-12T06:20:23.872 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-12T06:20:23.872 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-12T06:20:23.938 DEBUG:teuthology.orchestra.run.smithi152:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-12T06:20:23.988 DEBUG:teuthology.orchestra.run.smithi152:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-12T06:20:24.029 DEBUG:teuthology.orchestra.run.smithi152:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-12T06:20:24.121 DEBUG:teuthology.orchestra.run.smithi152:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-12T06:20:24.155 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-12T06:20:24.155 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-12T06:20:24.222 DEBUG:teuthology.orchestra.run.smithi096:> sudo service rsyslog restart 2024-09-12T06:20:24.224 DEBUG:teuthology.orchestra.run.smithi152:> sudo service rsyslog restart 2024-09-12T06:20:24.278 INFO:teuthology.orchestra.run.smithi096.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-12T06:20:24.308 INFO:teuthology.orchestra.run.smithi152.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-12T06:20:24.768 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-12T06:20:24.776 INFO:teuthology.task.internal:Starting timer... 2024-09-12T06:20:24.776 INFO:teuthology.run_tasks:Running task pcp... 2024-09-12T06:20:24.796 INFO:teuthology.run_tasks:Running task selinux... 2024-09-12T06:20:24.804 DEBUG:teuthology.orchestra.run.smithi096:> sudo service auditd rotate 2024-09-12T06:20:24.875 INFO:teuthology.orchestra.run.smithi096.stdout:Rotating logs: 2024-09-12T06:20:24.876 DEBUG:teuthology.orchestra.run.smithi152:> sudo service auditd rotate 2024-09-12T06:20:24.945 INFO:teuthology.orchestra.run.smithi152.stdout:Rotating logs: 2024-09-12T06:20:24.947 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-12T06:20:24.947 DEBUG:teuthology.orchestra.run.smithi096:> /usr/sbin/getenforce 2024-09-12T06:20:24.973 INFO:teuthology.orchestra.run.smithi096.stdout:Permissive 2024-09-12T06:20:24.973 DEBUG:teuthology.orchestra.run.smithi152:> /usr/sbin/getenforce 2024-09-12T06:20:25.001 INFO:teuthology.orchestra.run.smithi152.stdout:Permissive 2024-09-12T06:20:25.001 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi096.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi152.front.sepia.ceph.com': 'permissive'} 2024-09-12T06:20:25.002 DEBUG:teuthology.orchestra.run.smithi096:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-12T06:20:25.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-12T06:20:25.041 DEBUG:teuthology.orchestra.run.smithi152:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-12T06:20:25.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-12T06:20:25.070 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-12T06:20:25.070 DEBUG:teuthology.orchestra.run.smithi096:> sudo /usr/sbin/setenforce permissive 2024-09-12T06:20:25.113 DEBUG:teuthology.orchestra.run.smithi152:> sudo /usr/sbin/setenforce permissive 2024-09-12T06:20:25.146 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-12T06:20:25.156 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-12T06:20:25.159 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-12T06:20:25.279 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-12T06:20:25.290 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-12T06:20:25.291 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi096.front.sepia.ceph.com,smithi152.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-12T06:24:09.190 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi096.front.sepia.ceph.com'), Remote(name='ubuntu@smithi152.front.sepia.ceph.com')] 2024-09-12T06:24:09.192 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi096.front.sepia.ceph.com' 2024-09-12T06:24:09.193 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi096.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-12T06:24:09.279 DEBUG:teuthology.orchestra.run.smithi096:> true 2024-09-12T06:24:09.355 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi096.front.sepia.ceph.com' 2024-09-12T06:24:09.356 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi152.front.sepia.ceph.com' 2024-09-12T06:24:09.356 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-12T06:24:09.453 DEBUG:teuthology.orchestra.run.smithi152:> true 2024-09-12T06:24:09.528 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi152.front.sepia.ceph.com' 2024-09-12T06:24:09.528 INFO:teuthology.run_tasks:Running task clock... 2024-09-12T06:24:09.539 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-12T06:24:09.539 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-12T06:24:09.540 DEBUG:teuthology.orchestra.run.smithi096:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-12T06:24:09.542 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-12T06:24:09.542 DEBUG:teuthology.orchestra.run.smithi152:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-12T06:24:09.567 INFO:teuthology.orchestra.run.smithi096.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-12T06:24:09.581 INFO:teuthology.orchestra.run.smithi096.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-12T06:24:09.597 INFO:teuthology.orchestra.run.smithi152.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-12T06:24:09.605 INFO:teuthology.orchestra.run.smithi096.stderr:sudo: ntpd: command not found 2024-09-12T06:24:09.611 INFO:teuthology.orchestra.run.smithi152.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-12T06:24:09.616 INFO:teuthology.orchestra.run.smithi096.stdout:506 Cannot talk to daemon 2024-09-12T06:24:09.630 INFO:teuthology.orchestra.run.smithi096.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-12T06:24:09.635 INFO:teuthology.orchestra.run.smithi152.stderr:sudo: ntpd: command not found 2024-09-12T06:24:09.645 INFO:teuthology.orchestra.run.smithi096.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-12T06:24:09.646 INFO:teuthology.orchestra.run.smithi152.stdout:506 Cannot talk to daemon 2024-09-12T06:24:09.660 INFO:teuthology.orchestra.run.smithi152.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-12T06:24:09.675 INFO:teuthology.orchestra.run.smithi152.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-12T06:24:09.702 INFO:teuthology.orchestra.run.smithi096.stderr:bash: line 1: ntpq: command not found 2024-09-12T06:24:09.706 INFO:teuthology.orchestra.run.smithi096.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-12T06:24:09.706 INFO:teuthology.orchestra.run.smithi096.stdout:=============================================================================== 2024-09-12T06:24:09.706 INFO:teuthology.orchestra.run.smithi096.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-12T06:24:09.706 INFO:teuthology.orchestra.run.smithi096.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-12T06:24:09.706 INFO:teuthology.orchestra.run.smithi096.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-12T06:24:09.706 INFO:teuthology.orchestra.run.smithi096.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-12T06:24:09.731 INFO:teuthology.orchestra.run.smithi152.stderr:bash: line 1: ntpq: command not found 2024-09-12T06:24:09.734 INFO:teuthology.orchestra.run.smithi152.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-12T06:24:09.734 INFO:teuthology.orchestra.run.smithi152.stdout:=============================================================================== 2024-09-12T06:24:09.734 INFO:teuthology.orchestra.run.smithi152.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-12T06:24:09.734 INFO:teuthology.orchestra.run.smithi152.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-12T06:24:09.734 INFO:teuthology.orchestra.run.smithi152.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-12T06:24:09.734 INFO:teuthology.orchestra.run.smithi152.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-12T06:24:09.735 INFO:teuthology.run_tasks:Running task install... 2024-09-12T06:24:09.743 DEBUG:teuthology.task.install:project ceph 2024-09-12T06:24:09.743 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'd61b08865d4d98738f45ff2e4806d7fb6d03eca1'}} 2024-09-12T06:24:09.743 DEBUG:teuthology.task.install:config {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': 'd61b08865d4d98738f45ff2e4806d7fb6d03eca1'} 2024-09-12T06:24:09.743 INFO:teuthology.task.install:Using flavor: default 2024-09-12T06:24:09.748 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-12T06:24:09.748 INFO:teuthology.task.install:extra packages: ['rbd-mirror'] 2024-09-12T06:24:09.749 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['rbd-mirror'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd61b08865d4d98738f45ff2e4806d7fb6d03eca1', 'tag': None, 'wait_for_package': False} 2024-09-12T06:24:09.749 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d61b08865d4d98738f45ff2e4806d7fb6d03eca1 2024-09-12T06:24:09.752 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['rbd-mirror'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd61b08865d4d98738f45ff2e4806d7fb6d03eca1', 'tag': None, 'wait_for_package': False} 2024-09-12T06:24:09.752 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d61b08865d4d98738f45ff2e4806d7fb6d03eca1 2024-09-12T06:24:09.905 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-dis-testing/d61b08865d4d98738f45ff2e4806d7fb6d03eca1/centos/9/flavors/default/ 2024-09-12T06:24:09.906 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4883.gd61b0886 2024-09-12T06:24:09.921 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-dis-testing/d61b08865d4d98738f45ff2e4806d7fb6d03eca1/centos/9/flavors/default/ 2024-09-12T06:24:09.921 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4883.gd61b0886 2024-09-12T06:24:10.028 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-dis-testing/d61b08865d4d98738f45ff2e4806d7fb6d03eca1/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-dis-testing/d61b08865d4d98738f45ff2e4806d7fb6d03eca1/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-dis-testing/d61b08865d4d98738f45ff2e4806d7fb6d03eca1/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-12T06:24:10.028 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-12T06:24:10.028 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-12T06:24:10.054 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-dis-testing/d61b08865d4d98738f45ff2e4806d7fb6d03eca1/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-dis-testing/d61b08865d4d98738f45ff2e4806d7fb6d03eca1/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-dis-testing/d61b08865d4d98738f45ff2e4806d7fb6d03eca1/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-12T06:24:10.054 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-12T06:24:10.054 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-12T06:24:10.055 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, rbd-mirror on remote rpm x86_64 2024-09-12T06:24:10.055 DEBUG:teuthology.orchestra.run.smithi096:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d61b08865d4d98738f45ff2e4806d7fb6d03eca1/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-12T06:24:10.082 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, rbd-mirror on remote rpm x86_64 2024-09-12T06:24:10.082 DEBUG:teuthology.orchestra.run.smithi152:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d61b08865d4d98738f45ff2e4806d7fb6d03eca1/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-12T06:24:10.121 DEBUG:teuthology.orchestra.run.smithi096:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-12T06:24:10.149 DEBUG:teuthology.orchestra.run.smithi152:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-12T06:24:10.195 DEBUG:teuthology.orchestra.run.smithi096:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-12T06:24:10.225 DEBUG:teuthology.orchestra.run.smithi152:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-12T06:24:10.260 INFO:teuthology.orchestra.run.smithi096.stdout:check_obsoletes = 1 2024-09-12T06:24:10.261 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum clean all 2024-09-12T06:24:10.288 INFO:teuthology.orchestra.run.smithi152.stdout:check_obsoletes = 1 2024-09-12T06:24:10.290 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum clean all 2024-09-12T06:24:10.496 INFO:teuthology.orchestra.run.smithi096.stdout:45 files removed 2024-09-12T06:24:10.523 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-radosgw 2024-09-12T06:24:10.526 INFO:teuthology.orchestra.run.smithi152.stdout:45 files removed 2024-09-12T06:24:10.554 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-radosgw 2024-09-12T06:24:11.079 INFO:teuthology.orchestra.run.smithi096.stdout:ceph packages for x86_64 310 kB/s | 84 kB 00:00 2024-09-12T06:24:11.112 INFO:teuthology.orchestra.run.smithi152.stdout:ceph packages for x86_64 307 kB/s | 84 kB 00:00 2024-09-12T06:24:11.363 INFO:teuthology.orchestra.run.smithi096.stdout:ceph noarch packages 100 kB/s | 24 kB 00:00 2024-09-12T06:24:11.404 INFO:teuthology.orchestra.run.smithi152.stdout:ceph noarch packages 97 kB/s | 24 kB 00:00 2024-09-12T06:24:11.638 INFO:teuthology.orchestra.run.smithi096.stdout:ceph source packages 8.1 kB/s | 1.9 kB 00:00 2024-09-12T06:24:11.680 INFO:teuthology.orchestra.run.smithi152.stdout:ceph source packages 8.1 kB/s | 1.9 kB 00:00 2024-09-12T06:24:13.381 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - BaseOS 4.8 MB/s | 8.2 MB 00:01 2024-09-12T06:24:15.030 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - BaseOS 2.5 MB/s | 8.2 MB 00:03 2024-09-12T06:24:17.014 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - AppStream 23 MB/s | 20 MB 00:00 2024-09-12T06:24:22.532 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - CRB 3.7 MB/s | 6.5 MB 00:01 2024-09-12T06:24:24.124 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - Extras packages 76 kB/s | 19 kB 00:00 2024-09-12T06:24:25.825 INFO:teuthology.orchestra.run.smithi152.stdout:Extra Packages for Enterprise Linux 14 MB/s | 23 MB 00:01 2024-09-12T06:24:29.238 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - AppStream 1.4 MB/s | 20 MB 00:14 2024-09-12T06:24:31.225 INFO:teuthology.orchestra.run.smithi152.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-09-12T06:24:33.568 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-09-12T06:24:33.580 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-12T06:24:33.580 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-09-12T06:24:33.580 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-12T06:24:33.580 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-09-12T06:24:33.580 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-radosgw x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 11 M 2024-09-12T06:24:33.580 INFO:teuthology.orchestra.run.smithi152.stdout:Upgrading: 2024-09-12T06:24:33.580 INFO:teuthology.orchestra.run.smithi152.stdout: librados2 x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 3.5 M 2024-09-12T06:24:33.580 INFO:teuthology.orchestra.run.smithi152.stdout: librbd1 x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 3.2 M 2024-09-12T06:24:33.580 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-09-12T06:24:33.580 INFO:teuthology.orchestra.run.smithi152.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-12T06:24:33.581 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-base x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 5.5 M 2024-09-12T06:24:33.581 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-common x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 22 M 2024-09-12T06:24:33.581 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-selinux x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 25 k 2024-09-12T06:24:33.581 INFO:teuthology.orchestra.run.smithi152.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-12T06:24:33.581 INFO:teuthology.orchestra.run.smithi152.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-12T06:24:33.581 INFO:teuthology.orchestra.run.smithi152.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-12T06:24:33.581 INFO:teuthology.orchestra.run.smithi152.stdout: libcephfs2 x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 755 k 2024-09-12T06:24:33.581 INFO:teuthology.orchestra.run.smithi152.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-12T06:24:33.581 INFO:teuthology.orchestra.run.smithi152.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-12T06:24:33.581 INFO:teuthology.orchestra.run.smithi152.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-12T06:24:33.581 INFO:teuthology.orchestra.run.smithi152.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-12T06:24:33.581 INFO:teuthology.orchestra.run.smithi152.stdout: libradosstriper1 x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 508 k 2024-09-12T06:24:33.582 INFO:teuthology.orchestra.run.smithi152.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-12T06:24:33.582 INFO:teuthology.orchestra.run.smithi152.stdout: librgw2 x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 5.5 M 2024-09-12T06:24:33.582 INFO:teuthology.orchestra.run.smithi152.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-12T06:24:33.582 INFO:teuthology.orchestra.run.smithi152.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-12T06:24:33.582 INFO:teuthology.orchestra.run.smithi152.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-12T06:24:33.582 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ceph-argparse x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 45 k 2024-09-12T06:24:33.582 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ceph-common x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 143 k 2024-09-12T06:24:33.582 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cephfs x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 162 k 2024-09-12T06:24:33.582 INFO:teuthology.orchestra.run.smithi152.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-12T06:24:33.582 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rados x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 321 k 2024-09-12T06:24:33.582 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rbd x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 307 k 2024-09-12T06:24:33.582 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rgw x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 99 k 2024-09-12T06:24:33.582 INFO:teuthology.orchestra.run.smithi152.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-12T06:24:33.583 INFO:teuthology.orchestra.run.smithi152.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-12T06:24:33.583 INFO:teuthology.orchestra.run.smithi152.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-12T06:24:33.583 INFO:teuthology.orchestra.run.smithi152.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-12T06:24:33.583 INFO:teuthology.orchestra.run.smithi152.stdout:Installing weak dependencies: 2024-09-12T06:24:33.583 INFO:teuthology.orchestra.run.smithi152.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-12T06:24:33.583 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:24:33.583 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-09-12T06:24:33.583 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-12T06:24:33.584 INFO:teuthology.orchestra.run.smithi152.stdout:Install 31 Packages 2024-09-12T06:24:33.584 INFO:teuthology.orchestra.run.smithi152.stdout:Upgrade 2 Packages 2024-09-12T06:24:33.584 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:24:33.584 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 62 M 2024-09-12T06:24:33.584 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-09-12T06:24:34.676 INFO:teuthology.orchestra.run.smithi152.stdout:(1/33): ceph-base-19.3.0-4883.gd61b0886.el9.x86 11 MB/s | 5.5 MB 00:00 2024-09-12T06:24:34.727 INFO:teuthology.orchestra.run.smithi152.stdout:(2/33): ceph-selinux-19.3.0-4883.gd61b0886.el9. 503 kB/s | 25 kB 00:00 2024-09-12T06:24:34.827 INFO:teuthology.orchestra.run.smithi152.stdout:(3/33): libcephfs2-19.3.0-4883.gd61b0886.el9.x8 7.4 MB/s | 755 kB 00:00 2024-09-12T06:24:34.927 INFO:teuthology.orchestra.run.smithi152.stdout:(4/33): libradosstriper1-19.3.0-4883.gd61b0886. 5.0 MB/s | 508 kB 00:00 2024-09-12T06:24:35.094 INFO:teuthology.orchestra.run.smithi152.stdout:(5/33): ceph-radosgw-19.3.0-4883.gd61b0886.el9. 12 MB/s | 11 MB 00:00 2024-09-12T06:24:35.155 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - CRB 3.0 MB/s | 6.5 MB 00:02 2024-09-12T06:24:35.319 INFO:teuthology.orchestra.run.smithi152.stdout:(6/33): ceph-common-19.3.0-4883.gd61b0886.el9.x 19 MB/s | 22 MB 00:01 2024-09-12T06:24:35.344 INFO:teuthology.orchestra.run.smithi152.stdout:(7/33): python3-ceph-argparse-19.3.0-4883.gd61b 180 kB/s | 45 kB 00:00 2024-09-12T06:24:35.378 INFO:teuthology.orchestra.run.smithi152.stdout:(8/33): python3-ceph-common-19.3.0-4883.gd61b08 2.4 MB/s | 143 kB 00:00 2024-09-12T06:24:35.403 INFO:teuthology.orchestra.run.smithi152.stdout:(9/33): python3-cephfs-19.3.0-4883.gd61b0886.el 2.7 MB/s | 162 kB 00:00 2024-09-12T06:24:35.428 INFO:teuthology.orchestra.run.smithi152.stdout:(10/33): python3-rados-19.3.0-4883.gd61b0886.el 6.3 MB/s | 321 kB 00:00 2024-09-12T06:24:35.454 INFO:teuthology.orchestra.run.smithi152.stdout:(11/33): python3-rbd-19.3.0-4883.gd61b0886.el9. 6.0 MB/s | 307 kB 00:00 2024-09-12T06:24:35.479 INFO:teuthology.orchestra.run.smithi152.stdout:(12/33): python3-rgw-19.3.0-4883.gd61b0886.el9. 1.9 MB/s | 99 kB 00:00 2024-09-12T06:24:35.563 INFO:teuthology.orchestra.run.smithi152.stdout:(13/33): librgw2-19.3.0-4883.gd61b0886.el9.x86_ 8.7 MB/s | 5.5 MB 00:00 2024-09-12T06:24:35.746 INFO:teuthology.orchestra.run.smithi152.stdout:(14/33): boost-program-options-1.75.0-8.el9.x86 366 kB/s | 107 kB 00:00 2024-09-12T06:24:35.797 INFO:teuthology.orchestra.run.smithi152.stdout:(15/33): libnbd-1.20.2-2.el9.x86_64.rpm 538 kB/s | 171 kB 00:00 2024-09-12T06:24:35.822 INFO:teuthology.orchestra.run.smithi152.stdout:(16/33): librabbitmq-0.11.0-7.el9.x86_64.rpm 602 kB/s | 45 kB 00:00 2024-09-12T06:24:35.856 INFO:teuthology.orchestra.run.smithi152.stdout:(17/33): libpmemobj-1.12.1-1.el9.x86_64.rpm 548 kB/s | 160 kB 00:00 2024-09-12T06:24:35.906 INFO:teuthology.orchestra.run.smithi152.stdout:(18/33): python3-prettytable-0.7.2-27.el9.noarc 850 kB/s | 42 kB 00:00 2024-09-12T06:24:35.939 INFO:teuthology.orchestra.run.smithi152.stdout:(19/33): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.4 MB/s | 292 kB 00:00 2024-09-12T06:24:35.998 INFO:teuthology.orchestra.run.smithi152.stdout:(20/33): librdkafka-1.6.1-102.el9.x86_64.rpm 3.2 MB/s | 662 kB 00:00 2024-09-12T06:24:36.023 INFO:teuthology.orchestra.run.smithi152.stdout:(21/33): qatlib-service-24.02.0-1.el9.x86_64.rp 430 kB/s | 36 kB 00:00 2024-09-12T06:24:36.057 INFO:teuthology.orchestra.run.smithi152.stdout:(22/33): qatlib-24.02.0-1.el9.x86_64.rpm 1.4 MB/s | 221 kB 00:00 2024-09-12T06:24:36.082 INFO:teuthology.orchestra.run.smithi152.stdout:(23/33): qatzip-libs-1.2.0-1.el9.x86_64.rpm 566 kB/s | 47 kB 00:00 2024-09-12T06:24:36.340 INFO:teuthology.orchestra.run.smithi152.stdout:(24/33): libarrow-doc-9.0.0-11.el9.noarch.rpm 101 kB/s | 26 kB 00:00 2024-09-12T06:24:36.566 INFO:teuthology.orchestra.run.smithi152.stdout:(25/33): gperftools-libs-2.9.1-3.el9.x86_64.rpm 567 kB/s | 308 kB 00:00 2024-09-12T06:24:36.683 INFO:teuthology.orchestra.run.smithi152.stdout:(26/33): liboath-2.6.7-2.el9.x86_64.rpm 142 kB/s | 49 kB 00:00 2024-09-12T06:24:36.900 INFO:teuthology.orchestra.run.smithi152.stdout:(27/33): libunwind-1.6.2-1.el9.x86_64.rpm 202 kB/s | 67 kB 00:00 2024-09-12T06:24:37.333 INFO:teuthology.orchestra.run.smithi152.stdout:(28/33): parquet-libs-9.0.0-11.el9.x86_64.rpm 1.3 MB/s | 839 kB 00:00 2024-09-12T06:24:37.401 INFO:teuthology.orchestra.run.smithi152.stdout:(29/33): libarrow-9.0.0-11.el9.x86_64.rpm 3.3 MB/s | 4.4 MB 00:01 2024-09-12T06:24:37.425 INFO:teuthology.orchestra.run.smithi152.stdout:(30/33): re2-20211101-20.el9.x86_64.rpm 364 kB/s | 191 kB 00:00 2024-09-12T06:24:37.597 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - Extras packages 17 kB/s | 19 kB 00:01 2024-09-12T06:24:37.676 INFO:teuthology.orchestra.run.smithi152.stdout:(31/33): librados2-19.3.0-4883.gd61b0886.el9.x8 13 MB/s | 3.5 MB 00:00 2024-09-12T06:24:37.743 INFO:teuthology.orchestra.run.smithi152.stdout:(32/33): librbd1-19.3.0-4883.gd61b0886.el9.x86_ 10 MB/s | 3.2 MB 00:00 2024-09-12T06:24:38.101 INFO:teuthology.orchestra.run.smithi152.stdout:(33/33): thrift-0.15.0-2.el9.x86_64.rpm 2.1 MB/s | 1.6 MB 00:00 2024-09-12T06:24:38.104 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-09-12T06:24:38.105 INFO:teuthology.orchestra.run.smithi152.stdout:Total 14 MB/s | 62 MB 00:04 2024-09-12T06:24:38.227 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-09-12T06:24:38.263 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-09-12T06:24:38.263 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-09-12T06:24:38.937 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-09-12T06:24:38.939 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-09-12T06:24:39.201 INFO:teuthology.orchestra.run.smithi096.stdout:Extra Packages for Enterprise Linux 15 MB/s | 23 MB 00:01 2024-09-12T06:24:40.468 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-09-12T06:24:40.561 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/35 2024-09-12T06:24:40.589 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/35 2024-09-12T06:24:40.615 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/35 2024-09-12T06:24:40.637 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-12T06:24:40.661 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-12T06:24:40.856 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-12T06:24:40.909 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-12T06:24:41.320 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/35 2024-09-12T06:24:41.382 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : librados2-2:19.3.0-4883.gd61b0886.el9.x86_64 7/35 2024-09-12T06:24:41.476 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librados2-2:19.3.0-4883.gd61b0886.el9.x86_64 7/35 2024-09-12T06:24:41.565 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-rados-2:19.3.0-4883.gd61b0886.el9.x86_64 8/35 2024-09-12T06:24:41.592 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libcephfs2-2:19.3.0-4883.gd61b0886.el9.x86_64 9/35 2024-09-12T06:24:41.656 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: libcephfs2-2:19.3.0-4883.gd61b0886.el9.x86_64 9/35 2024-09-12T06:24:41.693 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/35 2024-09-12T06:24:41.736 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-ceph-argparse-2:19.3.0-4883.gd61b0886.el9. 11/35 2024-09-12T06:24:41.811 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cephfs-2:19.3.0-4883.gd61b0886.el9.x86_64 12/35 2024-09-12T06:24:41.836 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libradosstriper1-2:19.3.0-4883.gd61b0886.el9.x86_6 13/35 2024-09-12T06:24:41.883 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4883.gd61b0886.el9.x86_6 13/35 2024-09-12T06:24:41.918 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/35 2024-09-12T06:24:42.029 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/35 2024-09-12T06:24:42.060 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/35 2024-09-12T06:24:42.112 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/35 2024-09-12T06:24:42.142 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/35 2024-09-12T06:24:42.171 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/35 2024-09-12T06:24:42.205 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-12T06:24:42.265 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/35 2024-09-12T06:24:42.609 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 22/35 2024-09-12T06:24:42.651 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : librbd1-2:19.3.0-4883.gd61b0886.el9.x86_64 23/35 2024-09-12T06:24:42.742 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librbd1-2:19.3.0-4883.gd61b0886.el9.x86_64 23/35 2024-09-12T06:24:42.772 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-rbd-2:19.3.0-4883.gd61b0886.el9.x86_64 24/35 2024-09-12T06:24:42.809 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 25/35 2024-09-12T06:24:43.323 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 26/35 2024-09-12T06:24:43.353 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : librgw2-2:19.3.0-4883.gd61b0886.el9.x86_64 27/35 2024-09-12T06:24:43.425 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librgw2-2:19.3.0-4883.gd61b0886.el9.x86_64 27/35 2024-09-12T06:24:43.502 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-rgw-2:19.3.0-4883.gd61b0886.el9.x86_64 28/35 2024-09-12T06:24:43.530 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-ceph-common-2:19.3.0-4883.gd61b0886.el9.x8 29/35 2024-09-12T06:24:44.773 INFO:teuthology.orchestra.run.smithi096.stdout:lab-extras 25 kB/s | 1.7 kB 00:00 2024-09-12T06:24:45.889 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-common-2:19.3.0-4883.gd61b0886.el9.x86_64 30/35 2024-09-12T06:24:45.927 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-common-2:19.3.0-4883.gd61b0886.el9.x86_64 30/35 2024-09-12T06:24:46.508 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-common-2:19.3.0-4883.gd61b0886.el9.x86_64 30/35 2024-09-12T06:24:46.546 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-base-2:19.3.0-4883.gd61b0886.el9.x86_64 31/35 2024-09-12T06:24:46.636 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-base-2:19.3.0-4883.gd61b0886.el9.x86_64 31/35 2024-09-12T06:24:46.637 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-12T06:24:46.637 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-12T06:24:46.637 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:24:46.672 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-selinux-2:19.3.0-4883.gd61b0886.el9.x86_64 32/35 2024-09-12T06:24:47.125 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-12T06:24:47.136 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-12T06:24:47.136 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repository Size 2024-09-12T06:24:47.137 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-12T06:24:47.137 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-09-12T06:24:47.137 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-radosgw x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 11 M 2024-09-12T06:24:47.137 INFO:teuthology.orchestra.run.smithi096.stdout:Upgrading: 2024-09-12T06:24:47.137 INFO:teuthology.orchestra.run.smithi096.stdout: librados2 x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 3.5 M 2024-09-12T06:24:47.137 INFO:teuthology.orchestra.run.smithi096.stdout: librbd1 x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 3.2 M 2024-09-12T06:24:47.137 INFO:teuthology.orchestra.run.smithi096.stdout:Installing dependencies: 2024-09-12T06:24:47.137 INFO:teuthology.orchestra.run.smithi096.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-12T06:24:47.137 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-base x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 5.5 M 2024-09-12T06:24:47.137 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-common x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 22 M 2024-09-12T06:24:47.137 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-selinux x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 25 k 2024-09-12T06:24:47.138 INFO:teuthology.orchestra.run.smithi096.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-12T06:24:47.138 INFO:teuthology.orchestra.run.smithi096.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-12T06:24:47.138 INFO:teuthology.orchestra.run.smithi096.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-12T06:24:47.138 INFO:teuthology.orchestra.run.smithi096.stdout: libcephfs2 x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 755 k 2024-09-12T06:24:47.138 INFO:teuthology.orchestra.run.smithi096.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-12T06:24:47.138 INFO:teuthology.orchestra.run.smithi096.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-12T06:24:47.138 INFO:teuthology.orchestra.run.smithi096.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-12T06:24:47.138 INFO:teuthology.orchestra.run.smithi096.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-12T06:24:47.138 INFO:teuthology.orchestra.run.smithi096.stdout: libradosstriper1 x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 508 k 2024-09-12T06:24:47.138 INFO:teuthology.orchestra.run.smithi096.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-12T06:24:47.138 INFO:teuthology.orchestra.run.smithi096.stdout: librgw2 x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 5.5 M 2024-09-12T06:24:47.138 INFO:teuthology.orchestra.run.smithi096.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-12T06:24:47.138 INFO:teuthology.orchestra.run.smithi096.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-12T06:24:47.139 INFO:teuthology.orchestra.run.smithi096.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-12T06:24:47.139 INFO:teuthology.orchestra.run.smithi096.stdout: python3-ceph-argparse x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 45 k 2024-09-12T06:24:47.139 INFO:teuthology.orchestra.run.smithi096.stdout: python3-ceph-common x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 143 k 2024-09-12T06:24:47.139 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cephfs x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 162 k 2024-09-12T06:24:47.139 INFO:teuthology.orchestra.run.smithi096.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-12T06:24:47.139 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rados x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 321 k 2024-09-12T06:24:47.139 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rbd x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 307 k 2024-09-12T06:24:47.139 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rgw x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 99 k 2024-09-12T06:24:47.139 INFO:teuthology.orchestra.run.smithi096.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-12T06:24:47.139 INFO:teuthology.orchestra.run.smithi096.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-12T06:24:47.139 INFO:teuthology.orchestra.run.smithi096.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-12T06:24:47.139 INFO:teuthology.orchestra.run.smithi096.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-12T06:24:47.139 INFO:teuthology.orchestra.run.smithi096.stdout:Installing weak dependencies: 2024-09-12T06:24:47.140 INFO:teuthology.orchestra.run.smithi096.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-12T06:24:47.140 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:24:47.140 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-09-12T06:24:47.140 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-12T06:24:47.140 INFO:teuthology.orchestra.run.smithi096.stdout:Install 31 Packages 2024-09-12T06:24:47.140 INFO:teuthology.orchestra.run.smithi096.stdout:Upgrade 2 Packages 2024-09-12T06:24:47.141 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:24:47.141 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 62 M 2024-09-12T06:24:47.141 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-09-12T06:24:47.991 INFO:teuthology.orchestra.run.smithi096.stdout:(1/33): ceph-base-19.3.0-4883.gd61b0886.el9.x86 12 MB/s | 5.5 MB 00:00 2024-09-12T06:24:48.041 INFO:teuthology.orchestra.run.smithi096.stdout:(2/33): ceph-selinux-19.3.0-4883.gd61b0886.el9. 504 kB/s | 25 kB 00:00 2024-09-12T06:24:48.142 INFO:teuthology.orchestra.run.smithi096.stdout:(3/33): libcephfs2-19.3.0-4883.gd61b0886.el9.x8 7.4 MB/s | 755 kB 00:00 2024-09-12T06:24:48.234 INFO:teuthology.orchestra.run.smithi096.stdout:(4/33): libradosstriper1-19.3.0-4883.gd61b0886. 5.4 MB/s | 508 kB 00:00 2024-09-12T06:24:48.526 INFO:teuthology.orchestra.run.smithi096.stdout:(5/33): ceph-radosgw-19.3.0-4883.gd61b0886.el9. 11 MB/s | 11 MB 00:01 2024-09-12T06:24:48.742 INFO:teuthology.orchestra.run.smithi096.stdout:(6/33): ceph-common-19.3.0-4883.gd61b0886.el9.x 18 MB/s | 22 MB 00:01 2024-09-12T06:24:48.768 INFO:teuthology.orchestra.run.smithi096.stdout:(7/33): python3-ceph-argparse-19.3.0-4883.gd61b 187 kB/s | 45 kB 00:00 2024-09-12T06:24:48.801 INFO:teuthology.orchestra.run.smithi096.stdout:(8/33): python3-ceph-common-19.3.0-4883.gd61b08 2.4 MB/s | 143 kB 00:00 2024-09-12T06:24:48.827 INFO:teuthology.orchestra.run.smithi096.stdout:(9/33): python3-cephfs-19.3.0-4883.gd61b0886.el 2.7 MB/s | 162 kB 00:00 2024-09-12T06:24:48.852 INFO:teuthology.orchestra.run.smithi096.stdout:(10/33): python3-rados-19.3.0-4883.gd61b0886.el 6.2 MB/s | 321 kB 00:00 2024-09-12T06:24:48.936 INFO:teuthology.orchestra.run.smithi096.stdout:(11/33): librgw2-19.3.0-4883.gd61b0886.el9.x86_ 7.9 MB/s | 5.5 MB 00:00 2024-09-12T06:24:48.961 INFO:teuthology.orchestra.run.smithi096.stdout:(12/33): python3-rgw-19.3.0-4883.gd61b0886.el9. 915 kB/s | 99 kB 00:00 2024-09-12T06:24:48.986 INFO:teuthology.orchestra.run.smithi096.stdout:(13/33): python3-rbd-19.3.0-4883.gd61b0886.el9. 1.9 MB/s | 307 kB 00:00 2024-09-12T06:24:49.103 INFO:teuthology.orchestra.run.smithi096.stdout:(14/33): boost-program-options-1.75.0-8.el9.x86 640 kB/s | 107 kB 00:00 2024-09-12T06:24:49.154 INFO:teuthology.orchestra.run.smithi096.stdout:(15/33): libpmemobj-1.12.1-1.el9.x86_64.rpm 958 kB/s | 160 kB 00:00 2024-09-12T06:24:49.179 INFO:teuthology.orchestra.run.smithi096.stdout:(16/33): libnbd-1.20.2-2.el9.x86_64.rpm 784 kB/s | 171 kB 00:00 2024-09-12T06:24:49.204 INFO:teuthology.orchestra.run.smithi096.stdout:(17/33): librabbitmq-0.11.0-7.el9.x86_64.rpm 449 kB/s | 45 kB 00:00 2024-09-12T06:24:49.255 INFO:teuthology.orchestra.run.smithi096.stdout:(18/33): python3-prettytable-0.7.2-27.el9.noarc 844 kB/s | 42 kB 00:00 2024-09-12T06:24:49.280 INFO:teuthology.orchestra.run.smithi096.stdout:(19/33): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.8 MB/s | 292 kB 00:00 2024-09-12T06:24:49.313 INFO:teuthology.orchestra.run.smithi096.stdout:(20/33): librdkafka-1.6.1-102.el9.x86_64.rpm 4.1 MB/s | 662 kB 00:00 2024-09-12T06:24:49.339 INFO:teuthology.orchestra.run.smithi096.stdout:(21/33): qatlib-service-24.02.0-1.el9.x86_64.rp 613 kB/s | 36 kB 00:00 2024-09-12T06:24:49.364 INFO:teuthology.orchestra.run.smithi096.stdout:(22/33): qatlib-24.02.0-1.el9.x86_64.rpm 2.0 MB/s | 221 kB 00:00 2024-09-12T06:24:49.389 INFO:teuthology.orchestra.run.smithi096.stdout:(23/33): qatzip-libs-1.2.0-1.el9.x86_64.rpm 628 kB/s | 47 kB 00:00 2024-09-12T06:24:49.731 INFO:teuthology.orchestra.run.smithi096.stdout:(24/33): libarrow-doc-9.0.0-11.el9.noarch.rpm 76 kB/s | 26 kB 00:00 2024-09-12T06:24:49.906 INFO:teuthology.orchestra.run.smithi096.stdout:(25/33): liboath-2.6.7-2.el9.x86_64.rpm 277 kB/s | 49 kB 00:00 2024-09-12T06:24:49.965 INFO:teuthology.orchestra.run.smithi096.stdout:(26/33): gperftools-libs-2.9.1-3.el9.x86_64.rpm 491 kB/s | 308 kB 00:00 2024-09-12T06:24:50.040 INFO:teuthology.orchestra.run.smithi096.stdout:(27/33): libunwind-1.6.2-1.el9.x86_64.rpm 504 kB/s | 67 kB 00:00 2024-09-12T06:24:50.224 INFO:teuthology.orchestra.run.smithi096.stdout:(28/33): parquet-libs-9.0.0-11.el9.x86_64.rpm 3.2 MB/s | 839 kB 00:00 2024-09-12T06:24:50.249 INFO:teuthology.orchestra.run.smithi096.stdout:(29/33): re2-20211101-20.el9.x86_64.rpm 918 kB/s | 191 kB 00:00 2024-09-12T06:24:50.316 INFO:teuthology.orchestra.run.smithi096.stdout:(30/33): libarrow-9.0.0-11.el9.x86_64.rpm 4.6 MB/s | 4.4 MB 00:00 2024-09-12T06:24:50.450 INFO:teuthology.orchestra.run.smithi096.stdout:(31/33): thrift-0.15.0-2.el9.x86_64.rpm 7.0 MB/s | 1.6 MB 00:00 2024-09-12T06:24:50.533 INFO:teuthology.orchestra.run.smithi096.stdout:(32/33): librados2-19.3.0-4883.gd61b0886.el9.x8 12 MB/s | 3.5 MB 00:00 2024-09-12T06:24:50.617 INFO:teuthology.orchestra.run.smithi096.stdout:(33/33): librbd1-19.3.0-4883.gd61b0886.el9.x86_ 11 MB/s | 3.2 MB 00:00 2024-09-12T06:24:50.622 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-09-12T06:24:50.622 INFO:teuthology.orchestra.run.smithi096.stdout:Total 18 MB/s | 62 MB 00:03 2024-09-12T06:24:50.733 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-09-12T06:24:50.771 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-09-12T06:24:50.771 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-09-12T06:24:51.478 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-09-12T06:24:51.478 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-09-12T06:24:53.001 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-09-12T06:24:53.075 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/35 2024-09-12T06:24:53.105 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/35 2024-09-12T06:24:53.141 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/35 2024-09-12T06:24:53.161 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-12T06:24:53.182 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-12T06:24:53.390 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-12T06:24:53.442 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-12T06:24:53.774 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/35 2024-09-12T06:24:53.883 INFO:teuthology.orchestra.run.smithi096.stdout: Upgrading : librados2-2:19.3.0-4883.gd61b0886.el9.x86_64 7/35 2024-09-12T06:24:53.974 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: librados2-2:19.3.0-4883.gd61b0886.el9.x86_64 7/35 2024-09-12T06:24:54.062 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-rados-2:19.3.0-4883.gd61b0886.el9.x86_64 8/35 2024-09-12T06:24:54.085 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libcephfs2-2:19.3.0-4883.gd61b0886.el9.x86_64 9/35 2024-09-12T06:24:54.147 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: libcephfs2-2:19.3.0-4883.gd61b0886.el9.x86_64 9/35 2024-09-12T06:24:54.184 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/35 2024-09-12T06:24:54.228 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-ceph-argparse-2:19.3.0-4883.gd61b0886.el9. 11/35 2024-09-12T06:24:54.309 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cephfs-2:19.3.0-4883.gd61b0886.el9.x86_64 12/35 2024-09-12T06:24:54.337 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libradosstriper1-2:19.3.0-4883.gd61b0886.el9.x86_6 13/35 2024-09-12T06:24:54.401 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4883.gd61b0886.el9.x86_6 13/35 2024-09-12T06:24:54.436 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/35 2024-09-12T06:24:54.549 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/35 2024-09-12T06:24:54.579 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/35 2024-09-12T06:24:54.619 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/35 2024-09-12T06:24:54.660 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/35 2024-09-12T06:24:54.698 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/35 2024-09-12T06:24:54.740 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-12T06:24:54.777 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/35 2024-09-12T06:24:55.111 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 22/35 2024-09-12T06:24:55.153 INFO:teuthology.orchestra.run.smithi096.stdout: Upgrading : librbd1-2:19.3.0-4883.gd61b0886.el9.x86_64 23/35 2024-09-12T06:24:55.253 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: librbd1-2:19.3.0-4883.gd61b0886.el9.x86_64 23/35 2024-09-12T06:24:55.283 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-rbd-2:19.3.0-4883.gd61b0886.el9.x86_64 24/35 2024-09-12T06:24:55.319 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 25/35 2024-09-12T06:24:55.816 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 26/35 2024-09-12T06:24:55.839 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : librgw2-2:19.3.0-4883.gd61b0886.el9.x86_64 27/35 2024-09-12T06:24:55.910 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: librgw2-2:19.3.0-4883.gd61b0886.el9.x86_64 27/35 2024-09-12T06:24:55.988 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-rgw-2:19.3.0-4883.gd61b0886.el9.x86_64 28/35 2024-09-12T06:24:56.017 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-ceph-common-2:19.3.0-4883.gd61b0886.el9.x8 29/35 2024-09-12T06:24:56.849 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4883.gd61b0886.el9.x86_64 32/35 2024-09-12T06:24:56.849 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /sys 2024-09-12T06:24:56.850 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /proc 2024-09-12T06:24:56.850 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /mnt 2024-09-12T06:24:56.850 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /var/tmp 2024-09-12T06:24:56.850 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /home 2024-09-12T06:24:56.850 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /root 2024-09-12T06:24:56.850 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /tmp 2024-09-12T06:24:56.850 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:24:57.894 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-radosgw-2:19.3.0-4883.gd61b0886.el9.x86_64 33/35 2024-09-12T06:24:57.920 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4883.gd61b0886.el9.x86_64 33/35 2024-09-12T06:24:57.920 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-12T06:24:57.920 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-12T06:24:57.920 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-12T06:24:57.921 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-12T06:24:57.921 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:24:57.923 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-12T06:24:58.005 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-12T06:24:58.005 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-12T06:24:58.322 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-common-2:19.3.0-4883.gd61b0886.el9.x86_64 30/35 2024-09-12T06:24:58.355 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-common-2:19.3.0-4883.gd61b0886.el9.x86_64 30/35 2024-09-12T06:24:58.925 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-common-2:19.3.0-4883.gd61b0886.el9.x86_64 30/35 2024-09-12T06:24:58.971 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-base-2:19.3.0-4883.gd61b0886.el9.x86_64 31/35 2024-09-12T06:24:59.062 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-base-2:19.3.0-4883.gd61b0886.el9.x86_64 31/35 2024-09-12T06:24:59.062 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-12T06:24:59.062 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-12T06:24:59.063 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:24:59.105 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-selinux-2:19.3.0-4883.gd61b0886.el9.x86_64 32/35 2024-09-12T06:24:59.265 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-12T06:24:59.265 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-base-2:19.3.0-4883.gd61b0886.el9.x86_64 1/35 2024-09-12T06:24:59.265 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-common-2:19.3.0-4883.gd61b0886.el9.x86_64 2/35 2024-09-12T06:24:59.265 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-radosgw-2:19.3.0-4883.gd61b0886.el9.x86_64 3/35 2024-09-12T06:24:59.265 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-selinux-2:19.3.0-4883.gd61b0886.el9.x86_64 4/35 2024-09-12T06:24:59.266 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libcephfs2-2:19.3.0-4883.gd61b0886.el9.x86_64 5/35 2024-09-12T06:24:59.266 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libradosstriper1-2:19.3.0-4883.gd61b0886.el9.x86_6 6/35 2024-09-12T06:24:59.266 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librgw2-2:19.3.0-4883.gd61b0886.el9.x86_64 7/35 2024-09-12T06:24:59.266 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4883.gd61b0886.el9. 8/35 2024-09-12T06:24:59.266 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-ceph-common-2:19.3.0-4883.gd61b0886.el9.x8 9/35 2024-09-12T06:24:59.266 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cephfs-2:19.3.0-4883.gd61b0886.el9.x86_64 10/35 2024-09-12T06:24:59.266 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-rados-2:19.3.0-4883.gd61b0886.el9.x86_64 11/35 2024-09-12T06:24:59.266 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-rbd-2:19.3.0-4883.gd61b0886.el9.x86_64 12/35 2024-09-12T06:24:59.266 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-rgw-2:19.3.0-4883.gd61b0886.el9.x86_64 13/35 2024-09-12T06:24:59.266 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/35 2024-09-12T06:24:59.267 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 15/35 2024-09-12T06:24:59.267 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 16/35 2024-09-12T06:24:59.267 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 17/35 2024-09-12T06:24:59.267 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 18/35 2024-09-12T06:24:59.267 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 19/35 2024-09-12T06:24:59.267 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-12T06:24:59.267 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 21/35 2024-09-12T06:24:59.267 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 22/35 2024-09-12T06:24:59.267 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 23/35 2024-09-12T06:24:59.267 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 24/35 2024-09-12T06:24:59.268 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 25/35 2024-09-12T06:24:59.268 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 26/35 2024-09-12T06:24:59.268 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 27/35 2024-09-12T06:24:59.268 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 28/35 2024-09-12T06:24:59.268 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 29/35 2024-09-12T06:24:59.268 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : re2-1:20211101-20.el9.x86_64 30/35 2024-09-12T06:24:59.268 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 31/35 2024-09-12T06:24:59.268 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librados2-2:19.3.0-4883.gd61b0886.el9.x86_64 32/35 2024-09-12T06:24:59.268 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 33/35 2024-09-12T06:24:59.277 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librbd1-2:19.3.0-4883.gd61b0886.el9.x86_64 34/35 2024-09-12T06:25:00.245 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 35/35 2024-09-12T06:25:00.245 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:25:00.245 INFO:teuthology.orchestra.run.smithi152.stdout:Upgraded: 2024-09-12T06:25:00.245 INFO:teuthology.orchestra.run.smithi152.stdout: librados2-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T06:25:00.245 INFO:teuthology.orchestra.run.smithi152.stdout: librbd1-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T06:25:00.245 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-09-12T06:25:00.245 INFO:teuthology.orchestra.run.smithi152.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-12T06:25:00.245 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-base-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T06:25:00.245 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-common-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T06:25:00.245 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-radosgw-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T06:25:00.245 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-selinux-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T06:25:00.246 INFO:teuthology.orchestra.run.smithi152.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-12T06:25:00.246 INFO:teuthology.orchestra.run.smithi152.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-12T06:25:00.246 INFO:teuthology.orchestra.run.smithi152.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-12T06:25:00.246 INFO:teuthology.orchestra.run.smithi152.stdout: libcephfs2-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T06:25:00.246 INFO:teuthology.orchestra.run.smithi152.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-12T06:25:00.246 INFO:teuthology.orchestra.run.smithi152.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-12T06:25:00.246 INFO:teuthology.orchestra.run.smithi152.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-12T06:25:00.246 INFO:teuthology.orchestra.run.smithi152.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-12T06:25:00.246 INFO:teuthology.orchestra.run.smithi152.stdout: libradosstriper1-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T06:25:00.246 INFO:teuthology.orchestra.run.smithi152.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-12T06:25:00.246 INFO:teuthology.orchestra.run.smithi152.stdout: librgw2-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T06:25:00.246 INFO:teuthology.orchestra.run.smithi152.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-12T06:25:00.246 INFO:teuthology.orchestra.run.smithi152.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-12T06:25:00.246 INFO:teuthology.orchestra.run.smithi152.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-12T06:25:00.247 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ceph-argparse-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T06:25:00.247 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ceph-common-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T06:25:00.247 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cephfs-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T06:25:00.247 INFO:teuthology.orchestra.run.smithi152.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-12T06:25:00.247 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rados-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T06:25:00.247 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rbd-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T06:25:00.247 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rgw-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T06:25:00.247 INFO:teuthology.orchestra.run.smithi152.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-12T06:25:00.247 INFO:teuthology.orchestra.run.smithi152.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-12T06:25:00.247 INFO:teuthology.orchestra.run.smithi152.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-12T06:25:00.247 INFO:teuthology.orchestra.run.smithi152.stdout: re2-1:20211101-20.el9.x86_64 2024-09-12T06:25:00.247 INFO:teuthology.orchestra.run.smithi152.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-12T06:25:00.247 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:25:00.248 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-09-12T06:25:00.581 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-test 2024-09-12T06:25:01.142 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:00:30 ago on Thu 12 Sep 2024 06:24:31 AM UTC. 2024-09-12T06:25:01.283 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-09-12T06:25:01.283 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-12T06:25:01.284 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-09-12T06:25:01.284 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-12T06:25:01.284 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-09-12T06:25:01.284 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-test x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 50 M 2024-09-12T06:25:01.284 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-09-12T06:25:01.284 INFO:teuthology.orchestra.run.smithi152.stdout: libcephsqlite x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 161 k 2024-09-12T06:25:01.284 INFO:teuthology.orchestra.run.smithi152.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-12T06:25:01.284 INFO:teuthology.orchestra.run.smithi152.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-12T06:25:01.284 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:25:01.284 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-09-12T06:25:01.284 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-12T06:25:01.285 INFO:teuthology.orchestra.run.smithi152.stdout:Install 4 Packages 2024-09-12T06:25:01.285 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:25:01.285 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 51 M 2024-09-12T06:25:01.285 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 217 M 2024-09-12T06:25:01.285 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-09-12T06:25:01.577 INFO:teuthology.orchestra.run.smithi152.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.9 MB/s | 304 kB 00:00 2024-09-12T06:25:01.643 INFO:teuthology.orchestra.run.smithi152.stdout:(2/4): libcephsqlite-19.3.0-4883.gd61b0886.el9. 710 kB/s | 161 kB 00:00 2024-09-12T06:25:01.660 INFO:teuthology.orchestra.run.smithi152.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 765 kB/s | 64 kB 00:00 2024-09-12T06:25:02.969 INFO:teuthology.orchestra.run.smithi152.stdout:(4/4): ceph-test-19.3.0-4883.gd61b0886.el9.x86_ 32 MB/s | 50 MB 00:01 2024-09-12T06:25:02.971 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-09-12T06:25:02.971 INFO:teuthology.orchestra.run.smithi152.stdout:Total 30 MB/s | 51 MB 00:01 2024-09-12T06:25:02.999 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-09-12T06:25:03.019 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-09-12T06:25:03.019 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-09-12T06:25:03.426 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-09-12T06:25:03.427 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-09-12T06:25:03.959 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-09-12T06:25:04.045 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-12T06:25:04.084 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-12T06:25:04.112 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libcephsqlite-2:19.3.0-4883.gd61b0886.el9.x86_64 3/4 2024-09-12T06:25:08.876 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4883.gd61b0886.el9.x86_64 3/4 2024-09-12T06:25:08.952 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-test-2:19.3.0-4883.gd61b0886.el9.x86_64 4/4 2024-09-12T06:25:09.153 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4883.gd61b0886.el9.x86_64 32/35 2024-09-12T06:25:09.153 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /sys 2024-09-12T06:25:09.153 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /proc 2024-09-12T06:25:09.153 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /mnt 2024-09-12T06:25:09.153 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /var/tmp 2024-09-12T06:25:09.153 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /home 2024-09-12T06:25:09.154 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /root 2024-09-12T06:25:09.154 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /tmp 2024-09-12T06:25:09.154 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:25:10.137 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-radosgw-2:19.3.0-4883.gd61b0886.el9.x86_64 33/35 2024-09-12T06:25:10.163 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4883.gd61b0886.el9.x86_64 33/35 2024-09-12T06:25:10.163 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-12T06:25:10.163 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-12T06:25:10.163 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-12T06:25:10.163 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-12T06:25:10.163 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:25:10.165 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-12T06:25:10.249 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-12T06:25:10.249 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-12T06:25:11.511 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-test-2:19.3.0-4883.gd61b0886.el9.x86_64 4/4 2024-09-12T06:25:11.511 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-test-2:19.3.0-4883.gd61b0886.el9.x86_64 1/4 2024-09-12T06:25:11.511 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libcephsqlite-2:19.3.0-4883.gd61b0886.el9.x86_64 2/4 2024-09-12T06:25:11.511 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-12T06:25:11.538 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-12T06:25:11.538 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-base-2:19.3.0-4883.gd61b0886.el9.x86_64 1/35 2024-09-12T06:25:11.539 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-common-2:19.3.0-4883.gd61b0886.el9.x86_64 2/35 2024-09-12T06:25:11.539 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-radosgw-2:19.3.0-4883.gd61b0886.el9.x86_64 3/35 2024-09-12T06:25:11.539 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-selinux-2:19.3.0-4883.gd61b0886.el9.x86_64 4/35 2024-09-12T06:25:11.539 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libcephfs2-2:19.3.0-4883.gd61b0886.el9.x86_64 5/35 2024-09-12T06:25:11.539 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libradosstriper1-2:19.3.0-4883.gd61b0886.el9.x86_6 6/35 2024-09-12T06:25:11.539 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librgw2-2:19.3.0-4883.gd61b0886.el9.x86_64 7/35 2024-09-12T06:25:11.539 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4883.gd61b0886.el9. 8/35 2024-09-12T06:25:11.539 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-ceph-common-2:19.3.0-4883.gd61b0886.el9.x8 9/35 2024-09-12T06:25:11.539 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cephfs-2:19.3.0-4883.gd61b0886.el9.x86_64 10/35 2024-09-12T06:25:11.539 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-rados-2:19.3.0-4883.gd61b0886.el9.x86_64 11/35 2024-09-12T06:25:11.539 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-rbd-2:19.3.0-4883.gd61b0886.el9.x86_64 12/35 2024-09-12T06:25:11.539 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-rgw-2:19.3.0-4883.gd61b0886.el9.x86_64 13/35 2024-09-12T06:25:11.540 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/35 2024-09-12T06:25:11.540 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 15/35 2024-09-12T06:25:11.540 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 16/35 2024-09-12T06:25:11.540 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 17/35 2024-09-12T06:25:11.540 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 18/35 2024-09-12T06:25:11.540 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 19/35 2024-09-12T06:25:11.540 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-12T06:25:11.540 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 21/35 2024-09-12T06:25:11.540 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 22/35 2024-09-12T06:25:11.540 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 23/35 2024-09-12T06:25:11.540 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 24/35 2024-09-12T06:25:11.540 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 25/35 2024-09-12T06:25:11.541 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 26/35 2024-09-12T06:25:11.541 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 27/35 2024-09-12T06:25:11.541 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 28/35 2024-09-12T06:25:11.541 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 29/35 2024-09-12T06:25:11.541 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : re2-1:20211101-20.el9.x86_64 30/35 2024-09-12T06:25:11.542 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 31/35 2024-09-12T06:25:11.542 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librados2-2:19.3.0-4883.gd61b0886.el9.x86_64 32/35 2024-09-12T06:25:11.542 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 33/35 2024-09-12T06:25:11.542 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librbd1-2:19.3.0-4883.gd61b0886.el9.x86_64 34/35 2024-09-12T06:25:11.836 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-12T06:25:11.836 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:25:11.836 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-09-12T06:25:11.836 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-test-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T06:25:11.836 INFO:teuthology.orchestra.run.smithi152.stdout: libcephsqlite-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T06:25:11.836 INFO:teuthology.orchestra.run.smithi152.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-12T06:25:11.836 INFO:teuthology.orchestra.run.smithi152.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-12T06:25:11.836 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:25:11.836 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-09-12T06:25:12.107 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph 2024-09-12T06:25:12.389 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 35/35 2024-09-12T06:25:12.389 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:25:12.389 INFO:teuthology.orchestra.run.smithi096.stdout:Upgraded: 2024-09-12T06:25:12.389 INFO:teuthology.orchestra.run.smithi096.stdout: librados2-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T06:25:12.389 INFO:teuthology.orchestra.run.smithi096.stdout: librbd1-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T06:25:12.389 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-09-12T06:25:12.389 INFO:teuthology.orchestra.run.smithi096.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-12T06:25:12.390 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-base-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T06:25:12.390 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-common-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T06:25:12.390 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-radosgw-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T06:25:12.390 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-selinux-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T06:25:12.390 INFO:teuthology.orchestra.run.smithi096.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-12T06:25:12.390 INFO:teuthology.orchestra.run.smithi096.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-12T06:25:12.390 INFO:teuthology.orchestra.run.smithi096.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-12T06:25:12.390 INFO:teuthology.orchestra.run.smithi096.stdout: libcephfs2-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T06:25:12.390 INFO:teuthology.orchestra.run.smithi096.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-12T06:25:12.390 INFO:teuthology.orchestra.run.smithi096.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-12T06:25:12.390 INFO:teuthology.orchestra.run.smithi096.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-12T06:25:12.390 INFO:teuthology.orchestra.run.smithi096.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-12T06:25:12.390 INFO:teuthology.orchestra.run.smithi096.stdout: libradosstriper1-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T06:25:12.391 INFO:teuthology.orchestra.run.smithi096.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-12T06:25:12.391 INFO:teuthology.orchestra.run.smithi096.stdout: librgw2-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T06:25:12.391 INFO:teuthology.orchestra.run.smithi096.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-12T06:25:12.391 INFO:teuthology.orchestra.run.smithi096.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-12T06:25:12.391 INFO:teuthology.orchestra.run.smithi096.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-12T06:25:12.391 INFO:teuthology.orchestra.run.smithi096.stdout: python3-ceph-argparse-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T06:25:12.391 INFO:teuthology.orchestra.run.smithi096.stdout: python3-ceph-common-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T06:25:12.391 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cephfs-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T06:25:12.391 INFO:teuthology.orchestra.run.smithi096.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-12T06:25:12.391 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rados-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T06:25:12.391 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rbd-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T06:25:12.391 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rgw-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T06:25:12.391 INFO:teuthology.orchestra.run.smithi096.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-12T06:25:12.392 INFO:teuthology.orchestra.run.smithi096.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-12T06:25:12.392 INFO:teuthology.orchestra.run.smithi096.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-12T06:25:12.392 INFO:teuthology.orchestra.run.smithi096.stdout: re2-1:20211101-20.el9.x86_64 2024-09-12T06:25:12.392 INFO:teuthology.orchestra.run.smithi096.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-12T06:25:12.392 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:25:12.392 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-12T06:25:12.685 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:00:41 ago on Thu 12 Sep 2024 06:24:31 AM UTC. 2024-09-12T06:25:12.745 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-test 2024-09-12T06:25:12.834 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-09-12T06:25:12.837 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-12T06:25:12.837 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-09-12T06:25:12.837 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-12T06:25:12.837 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-09-12T06:25:12.837 INFO:teuthology.orchestra.run.smithi152.stdout: ceph x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 6.6 k 2024-09-12T06:25:12.838 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-09-12T06:25:12.838 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mds x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 2.4 M 2024-09-12T06:25:12.838 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 1.5 M 2024-09-12T06:25:12.838 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4883.gd61b0886.el9 ceph-noarch 291 k 2024-09-12T06:25:12.838 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mon x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 4.8 M 2024-09-12T06:25:12.838 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-osd x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 17 M 2024-09-12T06:25:12.838 INFO:teuthology.orchestra.run.smithi152.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-12T06:25:12.838 INFO:teuthology.orchestra.run.smithi152.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-12T06:25:12.838 INFO:teuthology.orchestra.run.smithi152.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-12T06:25:12.838 INFO:teuthology.orchestra.run.smithi152.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-12T06:25:12.838 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-12T06:25:12.838 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-12T06:25:12.838 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-12T06:25:12.839 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-12T06:25:12.839 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-12T06:25:12.839 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-12T06:25:12.839 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-collections 2024-09-12T06:25:12.839 INFO:teuthology.orchestra.run.smithi152.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-12T06:25:12.839 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-12T06:25:12.839 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-12T06:25:12.839 INFO:teuthology.orchestra.run.smithi152.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-12T06:25:12.839 INFO:teuthology.orchestra.run.smithi152.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-12T06:25:12.839 INFO:teuthology.orchestra.run.smithi152.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-12T06:25:12.839 INFO:teuthology.orchestra.run.smithi152.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-12T06:25:12.839 INFO:teuthology.orchestra.run.smithi152.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-12T06:25:12.839 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-12T06:25:12.840 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-12T06:25:12.840 INFO:teuthology.orchestra.run.smithi152.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-12T06:25:12.840 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-12T06:25:12.840 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-12T06:25:12.840 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-12T06:25:12.840 INFO:teuthology.orchestra.run.smithi152.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-12T06:25:12.840 INFO:teuthology.orchestra.run.smithi152.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-12T06:25:12.840 INFO:teuthology.orchestra.run.smithi152.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-12T06:25:12.840 INFO:teuthology.orchestra.run.smithi152.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-12T06:25:12.840 INFO:teuthology.orchestra.run.smithi152.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-12T06:25:12.840 INFO:teuthology.orchestra.run.smithi152.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-12T06:25:12.840 INFO:teuthology.orchestra.run.smithi152.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-12T06:25:12.841 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:25:12.841 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-09-12T06:25:12.841 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-12T06:25:12.841 INFO:teuthology.orchestra.run.smithi152.stdout:Install 37 Packages 2024-09-12T06:25:12.841 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:25:12.841 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 31 M 2024-09-12T06:25:12.841 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 109 M 2024-09-12T06:25:12.842 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-09-12T06:25:13.308 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:00:29 ago on Thu 12 Sep 2024 06:24:44 AM UTC. 2024-09-12T06:25:13.444 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-12T06:25:13.444 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-12T06:25:13.444 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repository Size 2024-09-12T06:25:13.445 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-12T06:25:13.445 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-09-12T06:25:13.445 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-test x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 50 M 2024-09-12T06:25:13.445 INFO:teuthology.orchestra.run.smithi096.stdout:Installing dependencies: 2024-09-12T06:25:13.445 INFO:teuthology.orchestra.run.smithi096.stdout: libcephsqlite x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 161 k 2024-09-12T06:25:13.445 INFO:teuthology.orchestra.run.smithi096.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-12T06:25:13.445 INFO:teuthology.orchestra.run.smithi096.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-12T06:25:13.445 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:25:13.445 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-09-12T06:25:13.445 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-12T06:25:13.445 INFO:teuthology.orchestra.run.smithi096.stdout:Install 4 Packages 2024-09-12T06:25:13.445 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:25:13.446 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 51 M 2024-09-12T06:25:13.446 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 217 M 2024-09-12T06:25:13.446 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-09-12T06:25:13.644 INFO:teuthology.orchestra.run.smithi152.stdout:(1/37): ceph-19.3.0-4883.gd61b0886.el9.x86_64.r 49 kB/s | 6.6 kB 00:00 2024-09-12T06:25:13.683 INFO:teuthology.orchestra.run.smithi096.stdout:(1/4): libcephsqlite-19.3.0-4883.gd61b0886.el9. 753 kB/s | 161 kB 00:00 2024-09-12T06:25:13.964 INFO:teuthology.orchestra.run.smithi152.stdout:(2/37): ceph-mds-19.3.0-4883.gd61b0886.el9.x86_ 5.2 MB/s | 2.4 MB 00:00 2024-09-12T06:25:14.048 INFO:teuthology.orchestra.run.smithi152.stdout:(3/37): ceph-mgr-19.3.0-4883.gd61b0886.el9.x86_ 2.8 MB/s | 1.5 MB 00:00 2024-09-12T06:25:14.156 INFO:teuthology.orchestra.run.smithi152.stdout:(4/37): ceph-mgr-modules-core-19.3.0-4883.gd61b 2.6 MB/s | 291 kB 00:00 2024-09-12T06:25:14.290 INFO:teuthology.orchestra.run.smithi152.stdout:(5/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.9 MB/s | 253 kB 00:00 2024-09-12T06:25:14.316 INFO:teuthology.orchestra.run.smithi096.stdout:(2/4): socat-1.7.4.1-6.el9.x86_64.rpm 359 kB/s | 304 kB 00:00 2024-09-12T06:25:14.350 INFO:teuthology.orchestra.run.smithi096.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 95 kB/s | 64 kB 00:00 2024-09-12T06:25:14.540 INFO:teuthology.orchestra.run.smithi152.stdout:(6/37): python3-cryptography-36.0.1-4.el9.x86_6 4.9 MB/s | 1.2 MB 00:00 2024-09-12T06:25:14.607 INFO:teuthology.orchestra.run.smithi152.stdout:(7/37): ceph-mon-19.3.0-4883.gd61b0886.el9.x86_ 5.0 MB/s | 4.8 MB 00:00 2024-09-12T06:25:14.632 INFO:teuthology.orchestra.run.smithi152.stdout:(8/37): python3-ply-3.11-14.el9.noarch.rpm 1.1 MB/s | 106 kB 00:00 2024-09-12T06:25:14.699 INFO:teuthology.orchestra.run.smithi152.stdout:(9/37): python3-requests-2.25.1-8.el9.noarch.rp 1.8 MB/s | 125 kB 00:00 2024-09-12T06:25:14.750 INFO:teuthology.orchestra.run.smithi152.stdout:(10/37): python3-pycparser-2.20-6.el9.noarch.rp 950 kB/s | 135 kB 00:00 2024-09-12T06:25:14.808 INFO:teuthology.orchestra.run.smithi152.stdout:(11/37): python3-urllib3-1.26.5-6.el9.noarch.rp 1.9 MB/s | 215 kB 00:00 2024-09-12T06:25:14.950 INFO:teuthology.orchestra.run.smithi152.stdout:(12/37): lua-5.4.4-4.el9.x86_64.rpm 941 kB/s | 188 kB 00:00 2024-09-12T06:25:14.992 INFO:teuthology.orchestra.run.smithi152.stdout:(13/37): python3-markupsafe-1.1.1-12.el9.x86_64 833 kB/s | 35 kB 00:00 2024-09-12T06:25:15.017 INFO:teuthology.orchestra.run.smithi152.stdout:(14/37): python3-mako-1.1.4-6.el9.noarch.rpm 824 kB/s | 172 kB 00:00 2024-09-12T06:25:15.042 INFO:teuthology.orchestra.run.smithi152.stdout:(15/37): python3-packaging-20.9-5.el9.noarch.rp 1.5 MB/s | 77 kB 00:00 2024-09-12T06:25:15.059 INFO:teuthology.orchestra.run.smithi152.stdout:(16/37): python3-pytz-2021.1-5.el9.noarch.rpm 1.2 MB/s | 51 kB 00:00 2024-09-12T06:25:15.236 INFO:teuthology.orchestra.run.smithi152.stdout:(17/37): ceph-osd-19.3.0-4883.gd61b0886.el9.x86 14 MB/s | 17 MB 00:01 2024-09-12T06:25:15.252 INFO:teuthology.orchestra.run.smithi152.stdout:(18/37): python3-toml-0.10.2-6.el9.noarch.rpm 199 kB/s | 42 kB 00:00 2024-09-12T06:25:15.360 INFO:teuthology.orchestra.run.smithi152.stdout:(19/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 402 kB/s | 43 kB 00:00 2024-09-12T06:25:15.383 INFO:teuthology.orchestra.run.smithi096.stdout:(4/4): ceph-test-19.3.0-4883.gd61b0886.el9.x86_ 26 MB/s | 50 MB 00:01 2024-09-12T06:25:15.385 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-09-12T06:25:15.386 INFO:teuthology.orchestra.run.smithi096.stdout:Total 26 MB/s | 51 MB 00:01 2024-09-12T06:25:15.411 INFO:teuthology.orchestra.run.smithi152.stdout:(20/37): luarocks-3.9.2-5.el9.noarch.rpm 865 kB/s | 151 kB 00:00 2024-09-12T06:25:15.414 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-09-12T06:25:15.435 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-09-12T06:25:15.435 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-09-12T06:25:15.461 INFO:teuthology.orchestra.run.smithi152.stdout:(21/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.7 MB/s | 172 kB 00:00 2024-09-12T06:25:15.495 INFO:teuthology.orchestra.run.smithi152.stdout:(22/37): python3-cherrypy-18.6.1-2.el9.noarch.r 4.2 MB/s | 358 kB 00:00 2024-09-12T06:25:15.520 INFO:teuthology.orchestra.run.smithi152.stdout:(23/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 183 kB/s | 11 kB 00:00 2024-09-12T06:25:15.553 INFO:teuthology.orchestra.run.smithi152.stdout:(24/37): python3-jaraco-classes-3.2.1-5.el9.noa 303 kB/s | 18 kB 00:00 2024-09-12T06:25:15.578 INFO:teuthology.orchestra.run.smithi152.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 395 kB/s | 23 kB 00:00 2024-09-12T06:25:15.604 INFO:teuthology.orchestra.run.smithi152.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 387 kB/s | 19 kB 00:00 2024-09-12T06:25:15.629 INFO:teuthology.orchestra.run.smithi152.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 393 kB/s | 20 kB 00:00 2024-09-12T06:25:15.654 INFO:teuthology.orchestra.run.smithi152.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 925 kB/s | 46 kB 00:00 2024-09-12T06:25:15.679 INFO:teuthology.orchestra.run.smithi152.stdout:(29/37): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-09-12T06:25:15.705 INFO:teuthology.orchestra.run.smithi152.stdout:(30/37): python3-pecan-1.4.2-3.el9.noarch.rpm 5.3 MB/s | 272 kB 00:00 2024-09-12T06:25:15.730 INFO:teuthology.orchestra.run.smithi152.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 327 kB/s | 16 kB 00:00 2024-09-12T06:25:15.755 INFO:teuthology.orchestra.run.smithi152.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.8 MB/s | 90 kB 00:00 2024-09-12T06:25:15.781 INFO:teuthology.orchestra.run.smithi152.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 712 kB/s | 36 kB 00:00 2024-09-12T06:25:15.806 INFO:teuthology.orchestra.run.smithi152.stdout:(34/37): python3-webob-1.8.8-2.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-09-12T06:25:15.835 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-09-12T06:25:15.836 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-09-12T06:25:15.849 INFO:teuthology.orchestra.run.smithi152.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 480 kB/s | 20 kB 00:00 2024-09-12T06:25:15.881 INFO:teuthology.orchestra.run.smithi152.stdout:(36/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.2 MB/s | 427 kB 00:00 2024-09-12T06:25:15.906 INFO:teuthology.orchestra.run.smithi152.stdout:(37/37): lua-devel-5.4.4-4.el9.x86_64.rpm 26 kB/s | 22 kB 00:00 2024-09-12T06:25:15.910 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-09-12T06:25:15.910 INFO:teuthology.orchestra.run.smithi152.stdout:Total 10 MB/s | 31 MB 00:03 2024-09-12T06:25:16.064 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-09-12T06:25:16.104 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-09-12T06:25:16.104 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-09-12T06:25:16.376 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-09-12T06:25:16.450 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-12T06:25:16.490 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-12T06:25:16.535 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libcephsqlite-2:19.3.0-4883.gd61b0886.el9.x86_64 3/4 2024-09-12T06:25:16.616 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-09-12T06:25:16.617 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-09-12T06:25:18.009 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-09-12T06:25:18.076 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-12T06:25:18.120 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-12T06:25:18.153 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-12T06:25:18.224 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-12T06:25:18.269 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-12T06:25:18.302 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-12T06:25:18.402 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-12T06:25:18.489 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-12T06:25:18.529 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-12T06:25:18.564 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-12T06:25:18.594 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-12T06:25:18.645 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-12T06:25:18.687 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-12T06:25:18.736 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-12T06:25:18.779 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-12T06:25:18.814 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-12T06:25:18.862 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-12T06:25:18.894 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-12T06:25:18.967 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-12T06:25:19.106 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-12T06:25:19.186 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-12T06:25:19.238 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-12T06:25:19.277 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-12T06:25:19.326 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-12T06:25:19.381 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-12T06:25:19.575 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-12T06:25:19.623 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-12T06:25:19.700 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-12T06:25:19.789 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-12T06:25:19.838 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-12T06:25:19.974 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-12T06:25:20.136 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4883.gd61b0886.el9. 32/37 2024-09-12T06:25:20.165 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-2:19.3.0-4883.gd61b0886.el9.x86_64 33/37 2024-09-12T06:25:20.192 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4883.gd61b0886.el9.x86_64 33/37 2024-09-12T06:25:20.192 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-12T06:25:20.192 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-12T06:25:20.192 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-12T06:25:20.192 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-12T06:25:20.192 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:25:21.320 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4883.gd61b0886.el9.x86_64 3/4 2024-09-12T06:25:21.400 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-test-2:19.3.0-4883.gd61b0886.el9.x86_64 4/4 2024-09-12T06:25:21.674 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-osd-2:19.3.0-4883.gd61b0886.el9.x86_64 34/37 2024-09-12T06:25:21.702 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-osd-2:19.3.0-4883.gd61b0886.el9.x86_64 34/37 2024-09-12T06:25:21.702 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-12T06:25:21.703 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-12T06:25:21.703 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-12T06:25:21.703 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-12T06:25:21.703 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:25:22.150 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mon-2:19.3.0-4883.gd61b0886.el9.x86_64 35/37 2024-09-12T06:25:22.174 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mon-2:19.3.0-4883.gd61b0886.el9.x86_64 35/37 2024-09-12T06:25:22.174 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-12T06:25:22.174 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-12T06:25:22.174 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-12T06:25:22.174 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-12T06:25:22.174 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:25:22.409 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mds-2:19.3.0-4883.gd61b0886.el9.x86_64 36/37 2024-09-12T06:25:22.433 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mds-2:19.3.0-4883.gd61b0886.el9.x86_64 36/37 2024-09-12T06:25:22.434 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-12T06:25:22.434 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-12T06:25:22.434 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-12T06:25:22.434 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-12T06:25:22.434 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:25:22.505 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-2:19.3.0-4883.gd61b0886.el9.x86_64 37/37 2024-09-12T06:25:23.880 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-test-2:19.3.0-4883.gd61b0886.el9.x86_64 4/4 2024-09-12T06:25:23.880 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-test-2:19.3.0-4883.gd61b0886.el9.x86_64 1/4 2024-09-12T06:25:23.880 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libcephsqlite-2:19.3.0-4883.gd61b0886.el9.x86_64 2/4 2024-09-12T06:25:23.880 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-12T06:25:24.222 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-12T06:25:24.222 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:25:24.223 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-09-12T06:25:24.223 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-test-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T06:25:24.223 INFO:teuthology.orchestra.run.smithi096.stdout: libcephsqlite-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T06:25:24.223 INFO:teuthology.orchestra.run.smithi096.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-12T06:25:24.223 INFO:teuthology.orchestra.run.smithi096.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-12T06:25:24.223 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:25:24.223 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-12T06:25:24.443 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-2:19.3.0-4883.gd61b0886.el9.x86_64 37/37 2024-09-12T06:25:24.443 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-2:19.3.0-4883.gd61b0886.el9.x86_64 1/37 2024-09-12T06:25:24.443 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mds-2:19.3.0-4883.gd61b0886.el9.x86_64 2/37 2024-09-12T06:25:24.444 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-2:19.3.0-4883.gd61b0886.el9.x86_64 3/37 2024-09-12T06:25:24.444 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mon-2:19.3.0-4883.gd61b0886.el9.x86_64 4/37 2024-09-12T06:25:24.444 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-osd-2:19.3.0-4883.gd61b0886.el9.x86_64 5/37 2024-09-12T06:25:24.444 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4883.gd61b0886.el9. 6/37 2024-09-12T06:25:24.444 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-12T06:25:24.444 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-12T06:25:24.444 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-12T06:25:24.444 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-12T06:25:24.444 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-12T06:25:24.444 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-12T06:25:24.444 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-12T06:25:24.444 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-12T06:25:24.444 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-12T06:25:24.445 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-12T06:25:24.445 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-12T06:25:24.445 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-12T06:25:24.445 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-12T06:25:24.445 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-12T06:25:24.445 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-12T06:25:24.445 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-12T06:25:24.445 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-12T06:25:24.445 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-12T06:25:24.445 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-12T06:25:24.445 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-12T06:25:24.445 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-12T06:25:24.446 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-12T06:25:24.447 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-12T06:25:24.447 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-12T06:25:24.447 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-12T06:25:24.447 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-12T06:25:24.447 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-12T06:25:24.447 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-12T06:25:24.447 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-12T06:25:24.447 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-12T06:25:24.502 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph 2024-09-12T06:25:25.072 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:00:41 ago on Thu 12 Sep 2024 06:24:44 AM UTC. 2024-09-12T06:25:25.215 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-12T06:25:25.217 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-12T06:25:25.218 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repository Size 2024-09-12T06:25:25.218 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-12T06:25:25.218 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-09-12T06:25:25.218 INFO:teuthology.orchestra.run.smithi096.stdout: ceph x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 6.6 k 2024-09-12T06:25:25.218 INFO:teuthology.orchestra.run.smithi096.stdout:Installing dependencies: 2024-09-12T06:25:25.218 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mds x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 2.4 M 2024-09-12T06:25:25.218 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 1.5 M 2024-09-12T06:25:25.218 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4883.gd61b0886.el9 ceph-noarch 291 k 2024-09-12T06:25:25.218 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mon x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 4.8 M 2024-09-12T06:25:25.218 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-osd x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 17 M 2024-09-12T06:25:25.218 INFO:teuthology.orchestra.run.smithi096.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-12T06:25:25.218 INFO:teuthology.orchestra.run.smithi096.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-12T06:25:25.218 INFO:teuthology.orchestra.run.smithi096.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-12T06:25:25.218 INFO:teuthology.orchestra.run.smithi096.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-12T06:25:25.219 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-12T06:25:25.219 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-12T06:25:25.219 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-12T06:25:25.219 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-12T06:25:25.219 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-12T06:25:25.219 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-12T06:25:25.219 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-collections 2024-09-12T06:25:25.219 INFO:teuthology.orchestra.run.smithi096.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-12T06:25:25.219 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-12T06:25:25.219 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-12T06:25:25.219 INFO:teuthology.orchestra.run.smithi096.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-12T06:25:25.219 INFO:teuthology.orchestra.run.smithi096.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-12T06:25:25.219 INFO:teuthology.orchestra.run.smithi096.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-12T06:25:25.220 INFO:teuthology.orchestra.run.smithi096.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-12T06:25:25.220 INFO:teuthology.orchestra.run.smithi096.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-12T06:25:25.220 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-12T06:25:25.220 INFO:teuthology.orchestra.run.smithi096.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-12T06:25:25.220 INFO:teuthology.orchestra.run.smithi096.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-12T06:25:25.220 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-12T06:25:25.220 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-12T06:25:25.220 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-12T06:25:25.220 INFO:teuthology.orchestra.run.smithi096.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-12T06:25:25.220 INFO:teuthology.orchestra.run.smithi096.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-12T06:25:25.220 INFO:teuthology.orchestra.run.smithi096.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-12T06:25:25.220 INFO:teuthology.orchestra.run.smithi096.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-12T06:25:25.220 INFO:teuthology.orchestra.run.smithi096.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-12T06:25:25.220 INFO:teuthology.orchestra.run.smithi096.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-12T06:25:25.221 INFO:teuthology.orchestra.run.smithi096.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-12T06:25:25.221 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:25:25.221 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-09-12T06:25:25.221 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-12T06:25:25.221 INFO:teuthology.orchestra.run.smithi096.stdout:Install 37 Packages 2024-09-12T06:25:25.221 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:25:25.222 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 31 M 2024-09-12T06:25:25.222 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 109 M 2024-09-12T06:25:25.222 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-09-12T06:25:25.603 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-12T06:25:25.603 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:25:25.603 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-09-12T06:25:25.603 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T06:25:25.603 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mds-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T06:25:25.603 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T06:25:25.603 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-modules-core-2:19.3.0-4883.gd61b0886.el9.noarch 2024-09-12T06:25:25.603 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mon-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T06:25:25.603 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-osd-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T06:25:25.603 INFO:teuthology.orchestra.run.smithi152.stdout: lua-5.4.4-4.el9.x86_64 2024-09-12T06:25:25.604 INFO:teuthology.orchestra.run.smithi152.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-12T06:25:25.604 INFO:teuthology.orchestra.run.smithi152.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-12T06:25:25.604 INFO:teuthology.orchestra.run.smithi152.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-12T06:25:25.604 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-12T06:25:25.604 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-12T06:25:25.604 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-12T06:25:25.604 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-12T06:25:25.604 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-12T06:25:25.604 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-12T06:25:25.604 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-12T06:25:25.604 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-12T06:25:25.604 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-12T06:25:25.604 INFO:teuthology.orchestra.run.smithi152.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-12T06:25:25.604 INFO:teuthology.orchestra.run.smithi152.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-12T06:25:25.605 INFO:teuthology.orchestra.run.smithi152.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-12T06:25:25.605 INFO:teuthology.orchestra.run.smithi152.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-12T06:25:25.605 INFO:teuthology.orchestra.run.smithi152.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-12T06:25:25.605 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-12T06:25:25.605 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ply-3.11-14.el9.noarch 2024-09-12T06:25:25.605 INFO:teuthology.orchestra.run.smithi152.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-12T06:25:25.605 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-12T06:25:25.605 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-12T06:25:25.605 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-12T06:25:25.605 INFO:teuthology.orchestra.run.smithi152.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-12T06:25:25.605 INFO:teuthology.orchestra.run.smithi152.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-12T06:25:25.605 INFO:teuthology.orchestra.run.smithi152.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-12T06:25:25.605 INFO:teuthology.orchestra.run.smithi152.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-12T06:25:25.606 INFO:teuthology.orchestra.run.smithi152.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-12T06:25:25.606 INFO:teuthology.orchestra.run.smithi152.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-12T06:25:25.606 INFO:teuthology.orchestra.run.smithi152.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-12T06:25:25.606 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:25:25.606 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-09-12T06:25:25.856 INFO:teuthology.orchestra.run.smithi096.stdout:(1/37): ceph-19.3.0-4883.gd61b0886.el9.x86_64.r 47 kB/s | 6.6 kB 00:00 2024-09-12T06:25:25.974 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-base 2024-09-12T06:25:26.031 INFO:teuthology.orchestra.run.smithi096.stdout:(2/37): ceph-mgr-19.3.0-4883.gd61b0886.el9.x86_ 4.9 MB/s | 1.5 MB 00:00 2024-09-12T06:25:26.090 INFO:teuthology.orchestra.run.smithi096.stdout:(3/37): ceph-mds-19.3.0-4883.gd61b0886.el9.x86_ 6.4 MB/s | 2.4 MB 00:00 2024-09-12T06:25:26.173 INFO:teuthology.orchestra.run.smithi096.stdout:(4/37): ceph-mgr-modules-core-19.3.0-4883.gd61b 3.4 MB/s | 291 kB 00:00 2024-09-12T06:25:26.240 INFO:teuthology.orchestra.run.smithi096.stdout:(5/37): ceph-mon-19.3.0-4883.gd61b0886.el9.x86_ 12 MB/s | 4.8 MB 00:00 2024-09-12T06:25:26.550 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:00:55 ago on Thu 12 Sep 2024 06:24:31 AM UTC. 2024-09-12T06:25:26.633 INFO:teuthology.orchestra.run.smithi096.stdout:(6/37): ceph-osd-19.3.0-4883.gd61b0886.el9.x86_ 29 MB/s | 17 MB 00:00 2024-09-12T06:25:26.640 INFO:teuthology.orchestra.run.smithi152.stdout:Package ceph-base-2:19.3.0-4883.gd61b0886.el9.x86_64 is already installed. 2024-09-12T06:25:26.682 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-09-12T06:25:26.683 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-09-12T06:25:26.683 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-09-12T06:25:26.749 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install cephadm 2024-09-12T06:25:26.924 INFO:teuthology.orchestra.run.smithi096.stdout:(7/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 337 kB/s | 253 kB 00:00 2024-09-12T06:25:27.050 INFO:teuthology.orchestra.run.smithi096.stdout:(8/37): python3-pycparser-2.20-6.el9.noarch.rpm 1.1 MB/s | 135 kB 00:00 2024-09-12T06:25:27.083 INFO:teuthology.orchestra.run.smithi096.stdout:(9/37): python3-ply-3.11-14.el9.noarch.rpm 236 kB/s | 106 kB 00:00 2024-09-12T06:25:27.167 INFO:teuthology.orchestra.run.smithi096.stdout:(10/37): python3-requests-2.25.1-8.el9.noarch.r 1.0 MB/s | 125 kB 00:00 2024-09-12T06:25:27.292 INFO:teuthology.orchestra.run.smithi096.stdout:(11/37): python3-cryptography-36.0.1-4.el9.x86_ 1.2 MB/s | 1.2 MB 00:01 2024-09-12T06:25:27.317 INFO:teuthology.orchestra.run.smithi096.stdout:(12/37): python3-urllib3-1.26.5-6.el9.noarch.rp 918 kB/s | 215 kB 00:00 2024-09-12T06:25:27.321 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:00:56 ago on Thu 12 Sep 2024 06:24:31 AM UTC. 2024-09-12T06:25:27.376 INFO:teuthology.orchestra.run.smithi096.stdout:(13/37): lua-5.4.4-4.el9.x86_64.rpm 902 kB/s | 188 kB 00:00 2024-09-12T06:25:27.426 INFO:teuthology.orchestra.run.smithi096.stdout:(14/37): python3-markupsafe-1.1.1-12.el9.x86_64 320 kB/s | 35 kB 00:00 2024-09-12T06:25:27.455 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-09-12T06:25:27.456 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-12T06:25:27.456 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-09-12T06:25:27.457 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-12T06:25:27.457 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-09-12T06:25:27.457 INFO:teuthology.orchestra.run.smithi152.stdout: cephadm noarch 2:19.3.0-4883.gd61b0886.el9 ceph-noarch 784 k 2024-09-12T06:25:27.457 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:25:27.457 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-09-12T06:25:27.457 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-12T06:25:27.458 INFO:teuthology.orchestra.run.smithi152.stdout:Install 1 Package 2024-09-12T06:25:27.458 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:25:27.458 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 784 k 2024-09-12T06:25:27.458 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 791 k 2024-09-12T06:25:27.458 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-09-12T06:25:27.472 INFO:teuthology.orchestra.run.smithi096.stdout:(15/37): python3-packaging-20.9-5.el9.noarch.rp 809 kB/s | 77 kB 00:00 2024-09-12T06:25:27.502 INFO:teuthology.orchestra.run.smithi096.stdout:(16/37): python3-mako-1.1.4-6.el9.noarch.rpm 822 kB/s | 172 kB 00:00 2024-09-12T06:25:27.527 INFO:teuthology.orchestra.run.smithi096.stdout:(17/37): python3-pytz-2021.1-5.el9.noarch.rpm 508 kB/s | 51 kB 00:00 2024-09-12T06:25:27.553 INFO:teuthology.orchestra.run.smithi096.stdout:(18/37): lua-devel-5.4.4-4.el9.x86_64.rpm 444 kB/s | 22 kB 00:00 2024-09-12T06:25:27.636 INFO:teuthology.orchestra.run.smithi096.stdout:(19/37): python3-toml-0.10.2-6.el9.noarch.rpm 253 kB/s | 42 kB 00:00 2024-09-12T06:25:27.742 INFO:teuthology.orchestra.run.smithi152.stdout:cephadm-19.3.0-4883.gd61b0886.el9.noarch.rpm 2.7 MB/s | 784 kB 00:00 2024-09-12T06:25:27.742 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-09-12T06:25:27.742 INFO:teuthology.orchestra.run.smithi152.stdout:Total 2.7 MB/s | 784 kB 00:00 2024-09-12T06:25:27.742 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-09-12T06:25:27.747 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-09-12T06:25:27.748 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-09-12T06:25:27.753 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-09-12T06:25:27.753 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-09-12T06:25:27.823 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-09-12T06:25:27.861 INFO:teuthology.orchestra.run.smithi096.stdout:(20/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 141 kB/s | 43 kB 00:00 2024-09-12T06:25:28.020 INFO:teuthology.orchestra.run.smithi096.stdout:(21/37): luarocks-3.9.2-5.el9.noarch.rpm 307 kB/s | 151 kB 00:00 2024-09-12T06:25:28.087 INFO:teuthology.orchestra.run.smithi096.stdout:(22/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 381 kB/s | 172 kB 00:00 2024-09-12T06:25:28.121 INFO:teuthology.orchestra.run.smithi096.stdout:(23/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 107 kB/s | 11 kB 00:00 2024-09-12T06:25:28.179 INFO:teuthology.orchestra.run.smithi096.stdout:(24/37): python3-jaraco-classes-3.2.1-5.el9.noa 192 kB/s | 18 kB 00:00 2024-09-12T06:25:28.205 INFO:teuthology.orchestra.run.smithi096.stdout:(25/37): python3-cherrypy-18.6.1-2.el9.noarch.r 1.0 MB/s | 358 kB 00:00 2024-09-12T06:25:28.228 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: cephadm-2:19.3.0-4883.gd61b0886.el9.noarch 1/1 2024-09-12T06:25:28.230 INFO:teuthology.orchestra.run.smithi096.stdout:(26/37): python3-jaraco-collections-3.0.0-8.el9 212 kB/s | 23 kB 00:00 2024-09-12T06:25:28.280 INFO:teuthology.orchestra.run.smithi096.stdout:(27/37): python3-jaraco-functools-3.5.0-2.el9.n 193 kB/s | 19 kB 00:00 2024-09-12T06:25:28.305 INFO:teuthology.orchestra.run.smithi096.stdout:(28/37): python3-jaraco-text-3.2.0-6.el9.noarch 196 kB/s | 20 kB 00:00 2024-09-12T06:25:28.328 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : cephadm-2:19.3.0-4883.gd61b0886.el9.noarch 1/1 2024-09-12T06:25:28.341 INFO:teuthology.orchestra.run.smithi096.stdout:(29/37): python3-logutils-0.3.5-21.el9.noarch.r 426 kB/s | 46 kB 00:00 2024-09-12T06:25:28.398 INFO:teuthology.orchestra.run.smithi096.stdout:(30/37): python3-more-itertools-8.12.0-2.el9.no 674 kB/s | 79 kB 00:00 2024-09-12T06:25:28.432 INFO:teuthology.orchestra.run.smithi096.stdout:(31/37): python3-pecan-1.4.2-3.el9.noarch.rpm 2.1 MB/s | 272 kB 00:00 2024-09-12T06:25:28.457 INFO:teuthology.orchestra.run.smithi096.stdout:(32/37): python3-portend-3.1.0-2.el9.noarch.rpm 140 kB/s | 16 kB 00:00 2024-09-12T06:25:28.516 INFO:teuthology.orchestra.run.smithi096.stdout:(33/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 766 kB/s | 90 kB 00:00 2024-09-12T06:25:28.540 INFO:teuthology.orchestra.run.smithi096.stdout:(34/37): python3-tempora-5.0.0-2.el9.noarch.rpm 329 kB/s | 36 kB 00:00 2024-09-12T06:25:28.583 INFO:teuthology.orchestra.run.smithi096.stdout:(35/37): python3-webob-1.8.8-2.el9.noarch.rpm 1.8 MB/s | 230 kB 00:00 2024-09-12T06:25:28.641 INFO:teuthology.orchestra.run.smithi096.stdout:(36/37): python3-zc-lockfile-2.0-10.el9.noarch. 199 kB/s | 20 kB 00:00 2024-09-12T06:25:28.686 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: cephadm-2:19.3.0-4883.gd61b0886.el9.noarch 1/1 2024-09-12T06:25:28.750 INFO:teuthology.orchestra.run.smithi096.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.8 MB/s | 427 kB 00:00 2024-09-12T06:25:28.751 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-09-12T06:25:28.751 INFO:teuthology.orchestra.run.smithi096.stdout:Total 8.7 MB/s | 31 MB 00:03 2024-09-12T06:25:28.905 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-09-12T06:25:28.943 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : cephadm-2:19.3.0-4883.gd61b0886.el9.noarch 1/1 2024-09-12T06:25:28.943 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:25:28.944 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-09-12T06:25:28.944 INFO:teuthology.orchestra.run.smithi152.stdout: cephadm-2:19.3.0-4883.gd61b0886.el9.noarch 2024-09-12T06:25:28.944 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:25:28.944 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-09-12T06:25:28.944 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-09-12T06:25:28.944 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-09-12T06:25:29.186 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-immutable-object-cache 2024-09-12T06:25:29.445 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-09-12T06:25:29.445 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-09-12T06:25:29.763 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:00:58 ago on Thu 12 Sep 2024 06:24:31 AM UTC. 2024-09-12T06:25:29.899 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-09-12T06:25:29.900 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-12T06:25:29.900 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repo Size 2024-09-12T06:25:29.900 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-12T06:25:29.900 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-09-12T06:25:29.900 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 145 k 2024-09-12T06:25:29.900 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:25:29.900 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-09-12T06:25:29.900 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-12T06:25:29.900 INFO:teuthology.orchestra.run.smithi152.stdout:Install 1 Package 2024-09-12T06:25:29.901 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:25:29.901 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 145 k 2024-09-12T06:25:29.901 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 432 k 2024-09-12T06:25:29.901 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-09-12T06:25:30.112 INFO:teuthology.orchestra.run.smithi152.stdout:ceph-immutable-object-cache-19.3.0-4883.gd61b08 685 kB/s | 145 kB 00:00 2024-09-12T06:25:30.113 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-09-12T06:25:30.113 INFO:teuthology.orchestra.run.smithi152.stdout:Total 679 kB/s | 145 kB 00:00 2024-09-12T06:25:30.114 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-09-12T06:25:30.121 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-09-12T06:25:30.121 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-09-12T06:25:30.179 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-09-12T06:25:30.179 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-09-12T06:25:30.334 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-09-12T06:25:30.430 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4883.gd61b0886. 1/1 2024-09-12T06:25:30.455 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4883.gd61b0886. 1/1 2024-09-12T06:25:30.455 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-12T06:25:30.455 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-12T06:25:30.455 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:25:30.832 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-09-12T06:25:30.902 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-12T06:25:30.943 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-12T06:25:30.980 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-12T06:25:31.053 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-12T06:25:31.093 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-12T06:25:31.134 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-12T06:25:31.227 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-12T06:25:31.305 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-12T06:25:31.352 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-12T06:25:31.397 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-12T06:25:31.405 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4883.gd61b0886. 1/1 2024-09-12T06:25:31.406 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:25:31.406 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-09-12T06:25:31.406 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-immutable-object-cache-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T06:25:31.406 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:25:31.406 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-09-12T06:25:31.427 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-12T06:25:31.462 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-12T06:25:31.495 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-12T06:25:31.536 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-12T06:25:31.579 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-12T06:25:31.622 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-12T06:25:31.656 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-mgr 2024-09-12T06:25:31.670 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-12T06:25:31.703 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-12T06:25:31.775 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-12T06:25:31.914 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-12T06:25:32.019 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-12T06:25:32.072 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-12T06:25:32.119 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-12T06:25:32.168 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-12T06:25:32.223 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-12T06:25:32.242 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:01 ago on Thu 12 Sep 2024 06:24:31 AM UTC. 2024-09-12T06:25:32.335 INFO:teuthology.orchestra.run.smithi152.stdout:Package ceph-mgr-2:19.3.0-4883.gd61b0886.el9.x86_64 is already installed. 2024-09-12T06:25:32.380 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-09-12T06:25:32.381 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-09-12T06:25:32.381 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-09-12T06:25:32.411 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-12T06:25:32.448 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-mgr-dashboard 2024-09-12T06:25:32.465 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-12T06:25:32.543 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-12T06:25:32.632 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-12T06:25:32.672 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-12T06:25:32.801 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-12T06:25:32.962 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4883.gd61b0886.el9. 32/37 2024-09-12T06:25:32.991 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mgr-2:19.3.0-4883.gd61b0886.el9.x86_64 33/37 2024-09-12T06:25:33.006 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:02 ago on Thu 12 Sep 2024 06:24:31 AM UTC. 2024-09-12T06:25:33.016 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4883.gd61b0886.el9.x86_64 33/37 2024-09-12T06:25:33.017 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-12T06:25:33.017 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-12T06:25:33.017 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-12T06:25:33.017 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-12T06:25:33.017 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:25:33.137 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-09-12T06:25:33.138 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-12T06:25:33.138 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-09-12T06:25:33.138 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-12T06:25:33.138 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-09-12T06:25:33.138 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4883.gd61b0886.el9 ceph-noarch 14 M 2024-09-12T06:25:33.138 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-09-12T06:25:33.138 INFO:teuthology.orchestra.run.smithi152.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-12T06:25:33.138 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4883.gd61b0886.el9 ceph-noarch 34 k 2024-09-12T06:25:33.139 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4883.gd61b0886.el9 ceph-noarch 16 k 2024-09-12T06:25:33.139 INFO:teuthology.orchestra.run.smithi152.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-12T06:25:33.139 INFO:teuthology.orchestra.run.smithi152.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-12T06:25:33.139 INFO:teuthology.orchestra.run.smithi152.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-12T06:25:33.139 INFO:teuthology.orchestra.run.smithi152.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-12T06:25:33.139 INFO:teuthology.orchestra.run.smithi152.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-12T06:25:33.139 INFO:teuthology.orchestra.run.smithi152.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-12T06:25:33.139 INFO:teuthology.orchestra.run.smithi152.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-12T06:25:33.139 INFO:teuthology.orchestra.run.smithi152.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-12T06:25:33.139 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:25:33.139 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-09-12T06:25:33.139 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-12T06:25:33.139 INFO:teuthology.orchestra.run.smithi152.stdout:Install 12 Packages 2024-09-12T06:25:33.139 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:25:33.140 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 19 M 2024-09-12T06:25:33.140 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 159 M 2024-09-12T06:25:33.140 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-09-12T06:25:33.982 INFO:teuthology.orchestra.run.smithi152.stdout:(1/12): ceph-prometheus-alerts-19.3.0-4883.gd61 100 kB/s | 16 kB 00:00 2024-09-12T06:25:34.007 INFO:teuthology.orchestra.run.smithi152.stdout:(2/12): ceph-grafana-dashboards-19.3.0-4883.gd6 180 kB/s | 34 kB 00:00 2024-09-12T06:25:34.182 INFO:teuthology.orchestra.run.smithi152.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.5 MB/s | 269 kB 00:00 2024-09-12T06:25:34.241 INFO:teuthology.orchestra.run.smithi152.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 3.9 MB/s | 1.0 MB 00:00 2024-09-12T06:25:34.475 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-osd-2:19.3.0-4883.gd61b0886.el9.x86_64 34/37 2024-09-12T06:25:34.502 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-osd-2:19.3.0-4883.gd61b0886.el9.x86_64 34/37 2024-09-12T06:25:34.502 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-12T06:25:34.502 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-12T06:25:34.502 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-12T06:25:34.502 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-12T06:25:34.503 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:25:34.508 INFO:teuthology.orchestra.run.smithi152.stdout:(5/12): ceph-mgr-dashboard-19.3.0-4883.gd61b088 20 MB/s | 14 MB 00:00 2024-09-12T06:25:34.533 INFO:teuthology.orchestra.run.smithi152.stdout:(6/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.4 MB/s | 863 kB 00:00 2024-09-12T06:25:34.767 INFO:teuthology.orchestra.run.smithi152.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 75 kB/s | 19 kB 00:00 2024-09-12T06:25:34.900 INFO:teuthology.orchestra.run.smithi152.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 832 kB/s | 548 kB 00:00 2024-09-12T06:25:34.951 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mon-2:19.3.0-4883.gd61b0886.el9.x86_64 35/37 2024-09-12T06:25:34.975 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mon-2:19.3.0-4883.gd61b0886.el9.x86_64 35/37 2024-09-12T06:25:34.975 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-12T06:25:34.975 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-12T06:25:34.975 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-12T06:25:34.975 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-12T06:25:34.975 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:25:35.159 INFO:teuthology.orchestra.run.smithi152.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 119 kB/s | 31 kB 00:00 2024-09-12T06:25:35.184 INFO:teuthology.orchestra.run.smithi152.stdout:(10/12): python3-grpcio-tools-1.46.7-10.el9.x86 346 kB/s | 144 kB 00:00 2024-09-12T06:25:35.202 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mds-2:19.3.0-4883.gd61b0886.el9.x86_64 36/37 2024-09-12T06:25:35.225 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mds-2:19.3.0-4883.gd61b0886.el9.x86_64 36/37 2024-09-12T06:25:35.225 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-12T06:25:35.225 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-12T06:25:35.225 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-12T06:25:35.225 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-12T06:25:35.225 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:25:35.289 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-2:19.3.0-4883.gd61b0886.el9.x86_64 37/37 2024-09-12T06:25:35.434 INFO:teuthology.orchestra.run.smithi152.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 2.3 MB/s | 2.0 MB 00:00 2024-09-12T06:25:35.651 INFO:teuthology.orchestra.run.smithi152.stdout:(12/12): python3-routes-2.5.1-5.el9.noarch.rpm 383 kB/s | 188 kB 00:00 2024-09-12T06:25:35.654 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-09-12T06:25:35.654 INFO:teuthology.orchestra.run.smithi152.stdout:Total 7.7 MB/s | 19 MB 00:02 2024-09-12T06:25:35.706 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-09-12T06:25:35.726 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-09-12T06:25:35.726 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-09-12T06:25:36.012 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-09-12T06:25:36.013 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-09-12T06:25:36.619 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-09-12T06:25:36.713 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-12T06:25:36.746 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-12T06:25:36.792 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-12T06:25:36.823 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-12T06:25:36.917 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-12T06:25:37.048 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-12T06:25:37.107 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-2:19.3.0-4883.gd61b0886.el9.x86_64 37/37 2024-09-12T06:25:37.107 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-2:19.3.0-4883.gd61b0886.el9.x86_64 1/37 2024-09-12T06:25:37.107 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mds-2:19.3.0-4883.gd61b0886.el9.x86_64 2/37 2024-09-12T06:25:37.107 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mgr-2:19.3.0-4883.gd61b0886.el9.x86_64 3/37 2024-09-12T06:25:37.107 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mon-2:19.3.0-4883.gd61b0886.el9.x86_64 4/37 2024-09-12T06:25:37.107 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-osd-2:19.3.0-4883.gd61b0886.el9.x86_64 5/37 2024-09-12T06:25:37.107 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4883.gd61b0886.el9. 6/37 2024-09-12T06:25:37.107 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-12T06:25:37.107 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-12T06:25:37.107 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-12T06:25:37.107 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-12T06:25:37.107 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-12T06:25:37.108 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-12T06:25:37.108 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-12T06:25:37.108 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-12T06:25:37.108 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-12T06:25:37.108 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-12T06:25:37.108 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-12T06:25:37.108 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-12T06:25:37.108 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-12T06:25:37.109 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-12T06:25:37.109 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-12T06:25:37.109 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-12T06:25:37.109 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-12T06:25:37.109 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-12T06:25:37.109 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-12T06:25:37.109 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-12T06:25:37.109 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-12T06:25:37.109 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-12T06:25:37.109 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-12T06:25:37.109 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-12T06:25:37.109 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-12T06:25:37.110 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-12T06:25:37.110 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-12T06:25:37.110 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-12T06:25:37.110 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-12T06:25:37.110 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-12T06:25:37.142 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-12T06:25:37.193 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-12T06:25:37.230 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-12T06:25:37.279 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4883.gd61b0886.el9 10/12 2024-09-12T06:25:38.155 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-12T06:25:38.155 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:25:38.156 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-09-12T06:25:38.156 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T06:25:38.156 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mds-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T06:25:38.156 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T06:25:38.156 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-modules-core-2:19.3.0-4883.gd61b0886.el9.noarch 2024-09-12T06:25:38.156 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mon-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T06:25:38.156 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-osd-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T06:25:38.156 INFO:teuthology.orchestra.run.smithi096.stdout: lua-5.4.4-4.el9.x86_64 2024-09-12T06:25:38.156 INFO:teuthology.orchestra.run.smithi096.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-12T06:25:38.156 INFO:teuthology.orchestra.run.smithi096.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-12T06:25:38.156 INFO:teuthology.orchestra.run.smithi096.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-12T06:25:38.156 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-12T06:25:38.157 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-12T06:25:38.157 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-12T06:25:38.157 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-12T06:25:38.157 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-12T06:25:38.157 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-12T06:25:38.157 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-12T06:25:38.157 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-12T06:25:38.157 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-12T06:25:38.157 INFO:teuthology.orchestra.run.smithi096.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-12T06:25:38.157 INFO:teuthology.orchestra.run.smithi096.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-12T06:25:38.157 INFO:teuthology.orchestra.run.smithi096.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-12T06:25:38.157 INFO:teuthology.orchestra.run.smithi096.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-12T06:25:38.157 INFO:teuthology.orchestra.run.smithi096.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-12T06:25:38.157 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-12T06:25:38.158 INFO:teuthology.orchestra.run.smithi096.stdout: python3-ply-3.11-14.el9.noarch 2024-09-12T06:25:38.158 INFO:teuthology.orchestra.run.smithi096.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-12T06:25:38.158 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-12T06:25:38.158 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-12T06:25:38.158 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-12T06:25:38.158 INFO:teuthology.orchestra.run.smithi096.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-12T06:25:38.158 INFO:teuthology.orchestra.run.smithi096.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-12T06:25:38.158 INFO:teuthology.orchestra.run.smithi096.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-12T06:25:38.158 INFO:teuthology.orchestra.run.smithi096.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-12T06:25:38.158 INFO:teuthology.orchestra.run.smithi096.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-12T06:25:38.158 INFO:teuthology.orchestra.run.smithi096.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-12T06:25:38.158 INFO:teuthology.orchestra.run.smithi096.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-12T06:25:38.158 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:25:38.158 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-12T06:25:38.587 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-base 2024-09-12T06:25:39.160 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:00:55 ago on Thu 12 Sep 2024 06:24:44 AM UTC. 2024-09-12T06:25:39.250 INFO:teuthology.orchestra.run.smithi096.stdout:Package ceph-base-2:19.3.0-4883.gd61b0886.el9.x86_64 is already installed. 2024-09-12T06:25:39.292 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-12T06:25:39.293 INFO:teuthology.orchestra.run.smithi096.stdout:Nothing to do. 2024-09-12T06:25:39.293 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-12T06:25:39.355 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install cephadm 2024-09-12T06:25:39.912 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:00:55 ago on Thu 12 Sep 2024 06:24:44 AM UTC. 2024-09-12T06:25:40.043 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-12T06:25:40.044 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-12T06:25:40.044 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repository Size 2024-09-12T06:25:40.044 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-12T06:25:40.044 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-09-12T06:25:40.044 INFO:teuthology.orchestra.run.smithi096.stdout: cephadm noarch 2:19.3.0-4883.gd61b0886.el9 ceph-noarch 784 k 2024-09-12T06:25:40.044 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:25:40.044 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-09-12T06:25:40.045 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-12T06:25:40.045 INFO:teuthology.orchestra.run.smithi096.stdout:Install 1 Package 2024-09-12T06:25:40.045 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:25:40.045 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 784 k 2024-09-12T06:25:40.045 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 791 k 2024-09-12T06:25:40.045 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-09-12T06:25:40.327 INFO:teuthology.orchestra.run.smithi096.stdout:cephadm-19.3.0-4883.gd61b0886.el9.noarch.rpm 2.7 MB/s | 784 kB 00:00 2024-09-12T06:25:40.327 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-09-12T06:25:40.328 INFO:teuthology.orchestra.run.smithi096.stdout:Total 2.7 MB/s | 784 kB 00:00 2024-09-12T06:25:40.328 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-09-12T06:25:40.333 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-09-12T06:25:40.333 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-09-12T06:25:40.339 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-09-12T06:25:40.340 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-09-12T06:25:40.413 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-09-12T06:25:40.454 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4883.gd61b0886.el 11/12 2024-09-12T06:25:40.516 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4883.gd61b0886.el9.noa 12/12 2024-09-12T06:25:40.822 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: cephadm-2:19.3.0-4883.gd61b0886.el9.noarch 1/1 2024-09-12T06:25:40.914 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : cephadm-2:19.3.0-4883.gd61b0886.el9.noarch 1/1 2024-09-12T06:25:41.285 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: cephadm-2:19.3.0-4883.gd61b0886.el9.noarch 1/1 2024-09-12T06:25:41.513 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : cephadm-2:19.3.0-4883.gd61b0886.el9.noarch 1/1 2024-09-12T06:25:41.513 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:25:41.513 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-09-12T06:25:41.513 INFO:teuthology.orchestra.run.smithi096.stdout: cephadm-2:19.3.0-4883.gd61b0886.el9.noarch 2024-09-12T06:25:41.513 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:25:41.513 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-12T06:25:41.730 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-immutable-object-cache 2024-09-12T06:25:42.315 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:00:58 ago on Thu 12 Sep 2024 06:24:44 AM UTC. 2024-09-12T06:25:42.451 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-12T06:25:42.452 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-12T06:25:42.452 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repo Size 2024-09-12T06:25:42.452 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-12T06:25:42.453 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-09-12T06:25:42.453 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 145 k 2024-09-12T06:25:42.453 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:25:42.453 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-09-12T06:25:42.453 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-12T06:25:42.453 INFO:teuthology.orchestra.run.smithi096.stdout:Install 1 Package 2024-09-12T06:25:42.453 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:25:42.453 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 145 k 2024-09-12T06:25:42.453 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 432 k 2024-09-12T06:25:42.453 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-09-12T06:25:42.507 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4883.gd61b0886.el9.noa 12/12 2024-09-12T06:25:42.508 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4883.gd61b0886.el 1/12 2024-09-12T06:25:42.508 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4883.gd61b0886.el9.noa 2/12 2024-09-12T06:25:42.508 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4883.gd61b0886.el9 3/12 2024-09-12T06:25:42.508 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-12T06:25:42.508 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-12T06:25:42.508 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-12T06:25:42.508 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-12T06:25:42.508 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-12T06:25:42.508 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-12T06:25:42.509 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-12T06:25:42.509 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-12T06:25:42.659 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-immutable-object-cache-19.3.0-4883.gd61b08 704 kB/s | 145 kB 00:00 2024-09-12T06:25:42.660 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-09-12T06:25:42.660 INFO:teuthology.orchestra.run.smithi096.stdout:Total 698 kB/s | 145 kB 00:00 2024-09-12T06:25:42.660 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-09-12T06:25:42.668 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-09-12T06:25:42.668 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-09-12T06:25:42.725 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-09-12T06:25:42.725 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-09-12T06:25:42.830 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-12T06:25:42.830 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:25:42.830 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-09-12T06:25:42.830 INFO:teuthology.orchestra.run.smithi152.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-12T06:25:42.830 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-grafana-dashboards-2:19.3.0-4883.gd61b0886.el9.noarch 2024-09-12T06:25:42.830 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-dashboard-2:19.3.0-4883.gd61b0886.el9.noarch 2024-09-12T06:25:42.830 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-prometheus-alerts-2:19.3.0-4883.gd61b0886.el9.noarch 2024-09-12T06:25:42.830 INFO:teuthology.orchestra.run.smithi152.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-12T06:25:42.831 INFO:teuthology.orchestra.run.smithi152.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-12T06:25:42.831 INFO:teuthology.orchestra.run.smithi152.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-12T06:25:42.831 INFO:teuthology.orchestra.run.smithi152.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-12T06:25:42.831 INFO:teuthology.orchestra.run.smithi152.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-12T06:25:42.831 INFO:teuthology.orchestra.run.smithi152.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-12T06:25:42.831 INFO:teuthology.orchestra.run.smithi152.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-12T06:25:42.831 INFO:teuthology.orchestra.run.smithi152.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-12T06:25:42.831 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:25:42.831 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-09-12T06:25:42.874 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-09-12T06:25:42.960 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4883.gd61b0886. 1/1 2024-09-12T06:25:42.985 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4883.gd61b0886. 1/1 2024-09-12T06:25:42.985 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-12T06:25:42.985 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-12T06:25:42.986 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:25:43.077 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-12T06:25:43.649 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:12 ago on Thu 12 Sep 2024 06:24:31 AM UTC. 2024-09-12T06:25:43.786 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-09-12T06:25:43.787 INFO:teuthology.orchestra.run.smithi152.stdout:======================================================================================= 2024-09-12T06:25:43.787 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-09-12T06:25:43.787 INFO:teuthology.orchestra.run.smithi152.stdout:======================================================================================= 2024-09-12T06:25:43.787 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-09-12T06:25:43.787 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4883.gd61b0886.el9 ceph-noarch 7.4 M 2024-09-12T06:25:43.787 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-09-12T06:25:43.788 INFO:teuthology.orchestra.run.smithi152.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-12T06:25:43.788 INFO:teuthology.orchestra.run.smithi152.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-12T06:25:43.788 INFO:teuthology.orchestra.run.smithi152.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-12T06:25:43.788 INFO:teuthology.orchestra.run.smithi152.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-12T06:25:43.788 INFO:teuthology.orchestra.run.smithi152.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-12T06:25:43.788 INFO:teuthology.orchestra.run.smithi152.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-12T06:25:43.788 INFO:teuthology.orchestra.run.smithi152.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-12T06:25:43.788 INFO:teuthology.orchestra.run.smithi152.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-12T06:25:43.788 INFO:teuthology.orchestra.run.smithi152.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-12T06:25:43.789 INFO:teuthology.orchestra.run.smithi152.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-12T06:25:43.789 INFO:teuthology.orchestra.run.smithi152.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-12T06:25:43.789 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:25:43.789 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-09-12T06:25:43.789 INFO:teuthology.orchestra.run.smithi152.stdout:======================================================================================= 2024-09-12T06:25:43.789 INFO:teuthology.orchestra.run.smithi152.stdout:Install 12 Packages 2024-09-12T06:25:43.789 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:25:43.790 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 38 M 2024-09-12T06:25:43.790 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 212 M 2024-09-12T06:25:43.790 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-09-12T06:25:43.912 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4883.gd61b0886. 1/1 2024-09-12T06:25:43.912 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:25:43.913 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-09-12T06:25:43.913 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-immutable-object-cache-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T06:25:43.913 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:25:43.913 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-12T06:25:44.129 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-mgr 2024-09-12T06:25:44.157 INFO:teuthology.orchestra.run.smithi152.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 985 kB/s | 191 kB 00:00 2024-09-12T06:25:44.224 INFO:teuthology.orchestra.run.smithi152.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 493 kB/s | 33 kB 00:00 2024-09-12T06:25:44.424 INFO:teuthology.orchestra.run.smithi152.stdout:(3/12): ceph-mgr-diskprediction-local-19.3.0-48 16 MB/s | 7.4 MB 00:00 2024-09-12T06:25:44.507 INFO:teuthology.orchestra.run.smithi152.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 219 kB/s | 18 kB 00:00 2024-09-12T06:25:44.558 INFO:teuthology.orchestra.run.smithi152.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 770 kB/s | 38 kB 00:00 2024-09-12T06:25:44.591 INFO:teuthology.orchestra.run.smithi152.stdout:(6/12): libgfortran-11.5.0-2.el9.x86_64.rpm 1.2 MB/s | 801 kB 00:00 2024-09-12T06:25:44.709 INFO:teuthology.orchestra.run.smithi152.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 2.0 MB/s | 245 kB 00:00 2024-09-12T06:25:44.710 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:00 ago on Thu 12 Sep 2024 06:24:44 AM UTC. 2024-09-12T06:25:44.803 INFO:teuthology.orchestra.run.smithi096.stdout:Package ceph-mgr-2:19.3.0-4883.gd61b0886.el9.x86_64 is already installed. 2024-09-12T06:25:44.846 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-12T06:25:44.847 INFO:teuthology.orchestra.run.smithi096.stdout:Nothing to do. 2024-09-12T06:25:44.847 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-12T06:25:44.921 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-mgr-dashboard 2024-09-12T06:25:45.009 INFO:teuthology.orchestra.run.smithi152.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 3.8 MB/s | 3.0 MB 00:00 2024-09-12T06:25:45.092 INFO:teuthology.orchestra.run.smithi152.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.9 MB/s | 250 kB 00:00 2024-09-12T06:25:45.476 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:01 ago on Thu 12 Sep 2024 06:24:44 AM UTC. 2024-09-12T06:25:45.607 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-12T06:25:45.609 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-12T06:25:45.609 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repository Size 2024-09-12T06:25:45.609 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-12T06:25:45.609 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-09-12T06:25:45.610 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4883.gd61b0886.el9 ceph-noarch 14 M 2024-09-12T06:25:45.610 INFO:teuthology.orchestra.run.smithi096.stdout:Installing dependencies: 2024-09-12T06:25:45.610 INFO:teuthology.orchestra.run.smithi096.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-12T06:25:45.610 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4883.gd61b0886.el9 ceph-noarch 34 k 2024-09-12T06:25:45.610 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4883.gd61b0886.el9 ceph-noarch 16 k 2024-09-12T06:25:45.610 INFO:teuthology.orchestra.run.smithi096.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-12T06:25:45.611 INFO:teuthology.orchestra.run.smithi096.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-12T06:25:45.611 INFO:teuthology.orchestra.run.smithi096.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-12T06:25:45.611 INFO:teuthology.orchestra.run.smithi096.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-12T06:25:45.611 INFO:teuthology.orchestra.run.smithi096.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-12T06:25:45.611 INFO:teuthology.orchestra.run.smithi096.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-12T06:25:45.611 INFO:teuthology.orchestra.run.smithi096.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-12T06:25:45.611 INFO:teuthology.orchestra.run.smithi096.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-12T06:25:45.612 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:25:45.612 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-09-12T06:25:45.612 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-12T06:25:45.612 INFO:teuthology.orchestra.run.smithi096.stdout:Install 12 Packages 2024-09-12T06:25:45.612 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:25:45.613 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 19 M 2024-09-12T06:25:45.613 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 159 M 2024-09-12T06:25:45.614 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-09-12T06:25:45.634 INFO:teuthology.orchestra.run.smithi152.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 4.6 MB/s | 4.9 MB 00:01 2024-09-12T06:25:45.989 INFO:teuthology.orchestra.run.smithi096.stdout:(1/12): ceph-prometheus-alerts-19.3.0-4883.gd61 109 kB/s | 16 kB 00:00 2024-09-12T06:25:46.005 INFO:teuthology.orchestra.run.smithi096.stdout:(2/12): ceph-grafana-dashboards-19.3.0-4883.gd6 205 kB/s | 34 kB 00:00 2024-09-12T06:25:46.035 INFO:teuthology.orchestra.run.smithi152.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 3.8 MB/s | 5.1 MB 00:01 2024-09-12T06:25:46.139 INFO:teuthology.orchestra.run.smithi096.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.0 MB/s | 269 kB 00:00 2024-09-12T06:25:46.198 INFO:teuthology.orchestra.run.smithi096.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.8 MB/s | 1.0 MB 00:00 2024-09-12T06:25:46.515 INFO:teuthology.orchestra.run.smithi096.stdout:(5/12): ceph-mgr-dashboard-19.3.0-4883.gd61b088 21 MB/s | 14 MB 00:00 2024-09-12T06:25:46.607 INFO:teuthology.orchestra.run.smithi096.stdout:(6/12): protobuf-compiler-3.14.0-13.el9.x86_64. 1.8 MB/s | 863 kB 00:00 2024-09-12T06:25:46.757 INFO:teuthology.orchestra.run.smithi096.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 80 kB/s | 19 kB 00:00 2024-09-12T06:25:46.782 INFO:teuthology.orchestra.run.smithi096.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 939 kB/s | 548 kB 00:00 2024-09-12T06:25:46.857 INFO:teuthology.orchestra.run.smithi096.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 410 kB/s | 31 kB 00:00 2024-09-12T06:25:46.949 INFO:teuthology.orchestra.run.smithi096.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.0 MB/s | 188 kB 00:00 2024-09-12T06:25:46.983 INFO:teuthology.orchestra.run.smithi096.stdout:(11/12): python3-grpcio-tools-1.46.7-10.el9.x86 640 kB/s | 144 kB 00:00 2024-09-12T06:25:47.202 INFO:teuthology.orchestra.run.smithi096.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 3.4 MB/s | 2.0 MB 00:00 2024-09-12T06:25:47.204 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-09-12T06:25:47.205 INFO:teuthology.orchestra.run.smithi096.stdout:Total 12 MB/s | 19 MB 00:01 2024-09-12T06:25:47.257 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-09-12T06:25:47.276 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-09-12T06:25:47.277 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-09-12T06:25:47.548 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-09-12T06:25:47.548 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-09-12T06:25:48.141 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-09-12T06:25:48.217 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-12T06:25:48.260 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-12T06:25:48.306 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-12T06:25:48.337 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-12T06:25:48.427 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-12T06:25:48.554 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-12T06:25:48.640 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-12T06:25:48.684 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-12T06:25:48.721 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-12T06:25:48.753 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4883.gd61b0886.el9 10/12 2024-09-12T06:25:49.435 INFO:teuthology.orchestra.run.smithi152.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 3.7 MB/s | 16 MB 00:04 2024-09-12T06:25:49.435 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-09-12T06:25:49.436 INFO:teuthology.orchestra.run.smithi152.stdout:Total 6.8 MB/s | 38 MB 00:05 2024-09-12T06:25:49.795 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-09-12T06:25:49.821 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-09-12T06:25:49.821 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-09-12T06:25:50.249 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-09-12T06:25:50.250 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-09-12T06:25:51.006 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-09-12T06:25:51.089 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-12T06:25:51.127 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-12T06:25:51.231 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-12T06:25:51.284 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-12T06:25:51.469 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-12T06:25:51.499 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-12T06:25:51.576 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-12T06:25:51.952 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4883.gd61b0886.el 11/12 2024-09-12T06:25:52.008 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4883.gd61b0886.el9.noa 12/12 2024-09-12T06:25:52.224 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-12T06:25:52.345 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-12T06:25:53.637 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-12T06:25:53.662 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4883.gd61b0886.el9.noa 12/12 2024-09-12T06:25:53.662 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4883.gd61b0886.el 1/12 2024-09-12T06:25:53.662 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4883.gd61b0886.el9.noa 2/12 2024-09-12T06:25:53.662 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4883.gd61b0886.el9 3/12 2024-09-12T06:25:53.662 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-12T06:25:53.662 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-12T06:25:53.663 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-12T06:25:53.663 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-12T06:25:53.663 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-12T06:25:53.663 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-12T06:25:53.663 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-12T06:25:53.663 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-12T06:25:53.995 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-12T06:25:53.995 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:25:53.996 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-09-12T06:25:53.996 INFO:teuthology.orchestra.run.smithi096.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-12T06:25:53.996 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-grafana-dashboards-2:19.3.0-4883.gd61b0886.el9.noarch 2024-09-12T06:25:53.996 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-dashboard-2:19.3.0-4883.gd61b0886.el9.noarch 2024-09-12T06:25:53.996 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-prometheus-alerts-2:19.3.0-4883.gd61b0886.el9.noarch 2024-09-12T06:25:53.996 INFO:teuthology.orchestra.run.smithi096.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-12T06:25:53.996 INFO:teuthology.orchestra.run.smithi096.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-12T06:25:53.996 INFO:teuthology.orchestra.run.smithi096.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-12T06:25:53.996 INFO:teuthology.orchestra.run.smithi096.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-12T06:25:53.996 INFO:teuthology.orchestra.run.smithi096.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-12T06:25:53.996 INFO:teuthology.orchestra.run.smithi096.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-12T06:25:53.996 INFO:teuthology.orchestra.run.smithi096.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-12T06:25:53.997 INFO:teuthology.orchestra.run.smithi096.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-12T06:25:53.997 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:25:53.997 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-12T06:25:54.262 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-12T06:25:54.692 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-12T06:25:54.722 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4883.gd61b0 12/12 2024-09-12T06:25:54.860 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:10 ago on Thu 12 Sep 2024 06:24:44 AM UTC. 2024-09-12T06:25:55.000 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-12T06:25:55.001 INFO:teuthology.orchestra.run.smithi096.stdout:======================================================================================= 2024-09-12T06:25:55.001 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repository Size 2024-09-12T06:25:55.001 INFO:teuthology.orchestra.run.smithi096.stdout:======================================================================================= 2024-09-12T06:25:55.002 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-09-12T06:25:55.002 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4883.gd61b0886.el9 ceph-noarch 7.4 M 2024-09-12T06:25:55.002 INFO:teuthology.orchestra.run.smithi096.stdout:Installing dependencies: 2024-09-12T06:25:55.002 INFO:teuthology.orchestra.run.smithi096.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-12T06:25:55.002 INFO:teuthology.orchestra.run.smithi096.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-12T06:25:55.002 INFO:teuthology.orchestra.run.smithi096.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-12T06:25:55.002 INFO:teuthology.orchestra.run.smithi096.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-12T06:25:55.002 INFO:teuthology.orchestra.run.smithi096.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-12T06:25:55.002 INFO:teuthology.orchestra.run.smithi096.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-12T06:25:55.002 INFO:teuthology.orchestra.run.smithi096.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-12T06:25:55.002 INFO:teuthology.orchestra.run.smithi096.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-12T06:25:55.002 INFO:teuthology.orchestra.run.smithi096.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-12T06:25:55.003 INFO:teuthology.orchestra.run.smithi096.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-12T06:25:55.003 INFO:teuthology.orchestra.run.smithi096.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-12T06:25:55.003 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:25:55.003 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-09-12T06:25:55.003 INFO:teuthology.orchestra.run.smithi096.stdout:======================================================================================= 2024-09-12T06:25:55.003 INFO:teuthology.orchestra.run.smithi096.stdout:Install 12 Packages 2024-09-12T06:25:55.003 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:25:55.004 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 38 M 2024-09-12T06:25:55.004 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 212 M 2024-09-12T06:25:55.004 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-09-12T06:25:55.283 INFO:teuthology.orchestra.run.smithi096.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.3 MB/s | 191 kB 00:00 2024-09-12T06:25:55.325 INFO:teuthology.orchestra.run.smithi096.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 4.3 MB/s | 801 kB 00:00 2024-09-12T06:25:55.625 INFO:teuthology.orchestra.run.smithi096.stdout:(3/12): ceph-mgr-diskprediction-local-19.3.0-48 15 MB/s | 7.4 MB 00:00 2024-09-12T06:25:55.650 INFO:teuthology.orchestra.run.smithi096.stdout:(4/12): flexiblas-3.0.4-8.el9.x86_64.rpm 89 kB/s | 33 kB 00:00 2024-09-12T06:25:55.851 INFO:teuthology.orchestra.run.smithi096.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 192 kB/s | 38 kB 00:00 2024-09-12T06:25:55.909 INFO:teuthology.orchestra.run.smithi096.stdout:(6/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 64 kB/s | 18 kB 00:00 2024-09-12T06:25:56.376 INFO:teuthology.orchestra.run.smithi096.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 524 kB/s | 245 kB 00:00 2024-09-12T06:25:56.560 INFO:teuthology.orchestra.run.smithi096.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.4 MB/s | 3.0 MB 00:01 2024-09-12T06:25:56.685 INFO:teuthology.orchestra.run.smithi096.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.0 MB/s | 250 kB 00:00 2024-09-12T06:25:57.302 INFO:teuthology.orchestra.run.smithi096.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 3.4 MB/s | 4.9 MB 00:01 2024-09-12T06:25:57.446 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4883.gd61b0 12/12 2024-09-12T06:25:57.447 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4883.gd61b0 1/12 2024-09-12T06:25:57.447 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-12T06:25:57.447 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-12T06:25:57.447 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-12T06:25:57.447 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-12T06:25:57.447 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-12T06:25:57.447 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-12T06:25:57.447 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-12T06:25:57.447 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-12T06:25:57.447 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-12T06:25:57.447 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-12T06:25:57.855 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-12T06:25:57.855 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:25:57.855 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-09-12T06:25:57.855 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4883.gd61b0886.el9.noarch 2024-09-12T06:25:57.855 INFO:teuthology.orchestra.run.smithi152.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-12T06:25:57.856 INFO:teuthology.orchestra.run.smithi152.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-12T06:25:57.856 INFO:teuthology.orchestra.run.smithi152.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-12T06:25:57.856 INFO:teuthology.orchestra.run.smithi152.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-12T06:25:57.856 INFO:teuthology.orchestra.run.smithi152.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-12T06:25:57.856 INFO:teuthology.orchestra.run.smithi152.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-12T06:25:57.856 INFO:teuthology.orchestra.run.smithi152.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-12T06:25:57.856 INFO:teuthology.orchestra.run.smithi152.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-12T06:25:57.856 INFO:teuthology.orchestra.run.smithi152.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-12T06:25:57.856 INFO:teuthology.orchestra.run.smithi152.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-12T06:25:57.856 INFO:teuthology.orchestra.run.smithi152.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-12T06:25:57.857 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:25:57.857 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-09-12T06:25:58.244 INFO:teuthology.orchestra.run.smithi096.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 2.7 MB/s | 5.1 MB 00:01 2024-09-12T06:25:58.309 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-mgr-rook 2024-09-12T06:25:58.889 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:27 ago on Thu 12 Sep 2024 06:24:31 AM UTC. 2024-09-12T06:25:59.026 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-09-12T06:25:59.027 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-12T06:25:59.027 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-09-12T06:25:59.027 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-12T06:25:59.027 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-09-12T06:25:59.027 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-rook noarch 2:19.3.0-4883.gd61b0886.el9 ceph-noarch 49 k 2024-09-12T06:25:59.027 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-09-12T06:25:59.027 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-12T06:25:59.027 INFO:teuthology.orchestra.run.smithi152.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-12T06:25:59.028 INFO:teuthology.orchestra.run.smithi152.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-12T06:25:59.028 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-12T06:25:59.028 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-12T06:25:59.028 INFO:teuthology.orchestra.run.smithi152.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-12T06:25:59.028 INFO:teuthology.orchestra.run.smithi152.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-12T06:25:59.028 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-12T06:25:59.028 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-12T06:25:59.028 INFO:teuthology.orchestra.run.smithi152.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-12T06:25:59.028 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-12T06:25:59.028 INFO:teuthology.orchestra.run.smithi152.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-12T06:25:59.028 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:25:59.028 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-09-12T06:25:59.028 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-12T06:25:59.029 INFO:teuthology.orchestra.run.smithi152.stdout:Install 13 Packages 2024-09-12T06:25:59.029 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:25:59.029 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 2.2 M 2024-09-12T06:25:59.029 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 26 M 2024-09-12T06:25:59.029 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-09-12T06:25:59.313 INFO:teuthology.orchestra.run.smithi152.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 213 kB/s | 19 kB 00:00 2024-09-12T06:25:59.330 INFO:teuthology.orchestra.run.smithi152.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 240 kB/s | 26 kB 00:00 2024-09-12T06:25:59.388 INFO:teuthology.orchestra.run.smithi152.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.9 MB/s | 222 kB 00:00 2024-09-12T06:25:59.413 INFO:teuthology.orchestra.run.smithi152.stdout:(4/13): ceph-mgr-rook-19.3.0-4883.gd61b0886.el9 258 kB/s | 49 kB 00:00 2024-09-12T06:25:59.447 INFO:teuthology.orchestra.run.smithi152.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.3 MB/s | 159 kB 00:00 2024-09-12T06:25:59.481 INFO:teuthology.orchestra.run.smithi152.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.0 MB/s | 279 kB 00:00 2024-09-12T06:25:59.548 INFO:teuthology.orchestra.run.smithi152.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 400 kB/s | 54 kB 00:00 2024-09-12T06:25:59.706 INFO:teuthology.orchestra.run.smithi152.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 124 kB/s | 32 kB 00:00 2024-09-12T06:25:59.731 INFO:teuthology.orchestra.run.smithi152.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 56 kB/s | 14 kB 00:00 2024-09-12T06:25:59.940 INFO:teuthology.orchestra.run.smithi152.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 284 kB/s | 59 kB 00:00 2024-09-12T06:26:00.023 INFO:teuthology.orchestra.run.smithi152.stdout:(11/13): python3-google-auth-2.34.0-1.el9.noarc 463 kB/s | 220 kB 00:00 2024-09-12T06:26:00.115 INFO:teuthology.orchestra.run.smithi152.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 512 kB/s | 90 kB 00:00 2024-09-12T06:26:00.182 INFO:teuthology.orchestra.run.smithi152.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 2.1 MB/s | 1.0 MB 00:00 2024-09-12T06:26:00.182 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-09-12T06:26:00.183 INFO:teuthology.orchestra.run.smithi152.stdout:Total 1.9 MB/s | 2.2 MB 00:01 2024-09-12T06:26:00.251 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-09-12T06:26:00.266 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-09-12T06:26:00.266 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-09-12T06:26:00.359 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-09-12T06:26:00.360 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-09-12T06:26:00.879 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-09-12T06:26:01.056 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-12T06:26:01.117 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-12T06:26:01.181 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-12T06:26:01.218 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-12T06:26:01.259 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-12T06:26:01.361 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-12T06:26:01.472 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-12T06:26:01.536 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-12T06:26:02.260 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-12T06:26:02.308 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-12T06:26:02.341 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-12T06:26:02.389 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-12T06:26:02.411 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-rook-2:19.3.0-4883.gd61b0886.el9.noarch 13/13 2024-09-12T06:26:03.206 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4883.gd61b0886.el9.noarch 13/13 2024-09-12T06:26:03.207 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4883.gd61b0886.el9.noarch 1/13 2024-09-12T06:26:03.207 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-12T06:26:03.207 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-12T06:26:03.207 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-12T06:26:03.207 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-12T06:26:03.207 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-12T06:26:03.207 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-12T06:26:03.207 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-12T06:26:03.207 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-12T06:26:03.207 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-12T06:26:03.207 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-12T06:26:03.207 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-12T06:26:03.552 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-12T06:26:03.552 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:26:03.553 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-09-12T06:26:03.553 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-rook-2:19.3.0-4883.gd61b0886.el9.noarch 2024-09-12T06:26:03.553 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-12T06:26:03.553 INFO:teuthology.orchestra.run.smithi152.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-12T06:26:03.553 INFO:teuthology.orchestra.run.smithi152.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-12T06:26:03.553 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-12T06:26:03.553 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-12T06:26:03.553 INFO:teuthology.orchestra.run.smithi152.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-12T06:26:03.553 INFO:teuthology.orchestra.run.smithi152.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-12T06:26:03.553 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-12T06:26:03.553 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-12T06:26:03.553 INFO:teuthology.orchestra.run.smithi152.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-12T06:26:03.553 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-12T06:26:03.554 INFO:teuthology.orchestra.run.smithi152.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-12T06:26:03.554 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:26:03.554 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-09-12T06:26:03.832 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-mgr-cephadm 2024-09-12T06:26:04.401 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:33 ago on Thu 12 Sep 2024 06:24:31 AM UTC. 2024-09-12T06:26:04.536 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-09-12T06:26:04.536 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-12T06:26:04.536 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-09-12T06:26:04.536 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-12T06:26:04.536 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-09-12T06:26:04.537 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4883.gd61b0886.el9 ceph-noarch 154 k 2024-09-12T06:26:04.537 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-09-12T06:26:04.537 INFO:teuthology.orchestra.run.smithi152.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-12T06:26:04.537 INFO:teuthology.orchestra.run.smithi152.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-12T06:26:04.537 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-12T06:26:04.537 INFO:teuthology.orchestra.run.smithi152.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-12T06:26:04.537 INFO:teuthology.orchestra.run.smithi152.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-12T06:26:04.537 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:26:04.537 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-09-12T06:26:04.537 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-12T06:26:04.538 INFO:teuthology.orchestra.run.smithi152.stdout:Install 6 Packages 2024-09-12T06:26:04.538 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:26:04.538 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 7.0 M 2024-09-12T06:26:04.538 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 33 M 2024-09-12T06:26:04.538 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-09-12T06:26:05.108 INFO:teuthology.orchestra.run.smithi152.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.3 MB/s | 248 kB 00:00 2024-09-12T06:26:05.133 INFO:teuthology.orchestra.run.smithi152.stdout:(2/6): ceph-mgr-cephadm-19.3.0-4883.gd61b0886.e 737 kB/s | 154 kB 00:00 2024-09-12T06:26:05.192 INFO:teuthology.orchestra.run.smithi152.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 994 kB/s | 58 kB 00:00 2024-09-12T06:26:05.234 INFO:teuthology.orchestra.run.smithi152.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 1.7 MB/s | 74 kB 00:00 2024-09-12T06:26:05.259 INFO:teuthology.orchestra.run.smithi152.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 3.6 MB/s | 548 kB 00:00 2024-09-12T06:26:05.334 INFO:teuthology.orchestra.run.smithi152.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 15 MB/s | 6.0 MB 00:00 2024-09-12T06:26:05.335 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-09-12T06:26:05.335 INFO:teuthology.orchestra.run.smithi152.stdout:Total 8.8 MB/s | 7.0 MB 00:00 2024-09-12T06:26:05.407 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-09-12T06:26:05.418 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-09-12T06:26:05.418 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-09-12T06:26:05.519 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-09-12T06:26:05.519 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-09-12T06:26:05.838 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-09-12T06:26:06.017 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-12T06:26:06.069 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-12T06:26:06.595 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-12T06:26:06.681 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-12T06:26:06.756 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-12T06:26:06.781 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4883.gd61b0886.el9.noarch 6/6 2024-09-12T06:26:07.569 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4883.gd61b0886.el9.noarch 6/6 2024-09-12T06:26:07.570 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4883.gd61b0886.el9.noarch 1/6 2024-09-12T06:26:07.570 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-12T06:26:07.570 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-12T06:26:07.570 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-12T06:26:07.570 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-12T06:26:07.858 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-12T06:26:07.859 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:26:07.859 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-09-12T06:26:07.859 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-cephadm-2:19.3.0-4883.gd61b0886.el9.noarch 2024-09-12T06:26:07.859 INFO:teuthology.orchestra.run.smithi152.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-12T06:26:07.859 INFO:teuthology.orchestra.run.smithi152.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-12T06:26:07.859 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-12T06:26:07.859 INFO:teuthology.orchestra.run.smithi152.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-12T06:26:07.859 INFO:teuthology.orchestra.run.smithi152.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-12T06:26:07.859 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:26:07.859 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-09-12T06:26:08.116 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-fuse 2024-09-12T06:26:08.686 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:37 ago on Thu 12 Sep 2024 06:24:31 AM UTC. 2024-09-12T06:26:08.821 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-09-12T06:26:08.822 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-12T06:26:08.822 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-09-12T06:26:08.822 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-12T06:26:08.822 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-09-12T06:26:08.822 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-fuse x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 900 k 2024-09-12T06:26:08.822 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-09-12T06:26:08.822 INFO:teuthology.orchestra.run.smithi152.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-12T06:26:08.823 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:26:08.823 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-09-12T06:26:08.823 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-12T06:26:08.823 INFO:teuthology.orchestra.run.smithi152.stdout:Install 2 Packages 2024-09-12T06:26:08.823 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:26:08.823 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 980 k 2024-09-12T06:26:08.823 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 2.9 M 2024-09-12T06:26:08.823 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-09-12T06:26:08.986 INFO:teuthology.orchestra.run.smithi152.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 562 kB/s | 80 kB 00:00 2024-09-12T06:26:09.127 INFO:teuthology.orchestra.run.smithi152.stdout:(2/2): ceph-fuse-19.3.0-4883.gd61b0886.el9.x86_ 3.1 MB/s | 900 kB 00:00 2024-09-12T06:26:09.128 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-09-12T06:26:09.128 INFO:teuthology.orchestra.run.smithi152.stdout:Total 3.1 MB/s | 980 kB 00:00 2024-09-12T06:26:09.140 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-09-12T06:26:09.163 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-09-12T06:26:09.163 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-09-12T06:26:09.245 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-09-12T06:26:09.246 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-09-12T06:26:09.430 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-09-12T06:26:09.572 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-12T06:26:09.643 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-fuse-2:19.3.0-4883.gd61b0886.el9.x86_64 2/2 2024-09-12T06:26:10.360 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4883.gd61b0886.el9.x86_64 2/2 2024-09-12T06:26:10.361 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-fuse-2:19.3.0-4883.gd61b0886.el9.x86_64 1/2 2024-09-12T06:26:10.630 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-12T06:26:10.630 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:26:10.630 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-09-12T06:26:10.630 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-fuse-2:19.3.0-4883.gd61b0886.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-12T06:26:10.631 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:26:10.631 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-09-12T06:26:10.917 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-volume 2024-09-12T06:26:11.503 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:40 ago on Thu 12 Sep 2024 06:24:31 AM UTC. 2024-09-12T06:26:11.645 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-09-12T06:26:11.646 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-12T06:26:11.646 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-09-12T06:26:11.646 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-12T06:26:11.646 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-09-12T06:26:11.646 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-volume noarch 2:19.3.0-4883.gd61b0886.el9 ceph-noarch 285 k 2024-09-12T06:26:11.646 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:26:11.646 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-09-12T06:26:11.647 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-12T06:26:11.647 INFO:teuthology.orchestra.run.smithi152.stdout:Install 1 Package 2024-09-12T06:26:11.647 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:26:11.647 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 285 k 2024-09-12T06:26:11.647 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 1.3 M 2024-09-12T06:26:11.647 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-09-12T06:26:11.873 INFO:teuthology.orchestra.run.smithi152.stdout:ceph-volume-19.3.0-4883.gd61b0886.el9.noarch.rp 1.2 MB/s | 285 kB 00:00 2024-09-12T06:26:11.874 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-09-12T06:26:11.874 INFO:teuthology.orchestra.run.smithi152.stdout:Total 1.2 MB/s | 285 kB 00:00 2024-09-12T06:26:11.874 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-09-12T06:26:11.883 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-09-12T06:26:11.883 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-09-12T06:26:11.909 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-09-12T06:26:11.909 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-09-12T06:26:12.136 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-09-12T06:26:12.209 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-volume-2:19.3.0-4883.gd61b0886.el9.noarch 1/1 2024-09-12T06:26:12.225 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-volume-2:19.3.0-4883.gd61b0886.el9.noarch 1/1 2024-09-12T06:26:12.225 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-12T06:26:12.225 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-12T06:26:12.226 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:26:13.250 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-volume-2:19.3.0-4883.gd61b0886.el9.noarch 1/1 2024-09-12T06:26:13.250 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:26:13.250 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-09-12T06:26:13.250 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-volume-2:19.3.0-4883.gd61b0886.el9.noarch 2024-09-12T06:26:13.250 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:26:13.250 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-09-12T06:26:13.519 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install librados-devel 2024-09-12T06:26:14.102 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:43 ago on Thu 12 Sep 2024 06:24:31 AM UTC. 2024-09-12T06:26:14.245 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-09-12T06:26:14.246 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-12T06:26:14.247 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repo Size 2024-09-12T06:26:14.247 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-12T06:26:14.247 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-09-12T06:26:14.247 INFO:teuthology.orchestra.run.smithi152.stdout: librados-devel x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 127 k 2024-09-12T06:26:14.247 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:26:14.247 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-09-12T06:26:14.247 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-12T06:26:14.247 INFO:teuthology.orchestra.run.smithi152.stdout:Install 1 Package 2024-09-12T06:26:14.247 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:26:14.247 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 127 k 2024-09-12T06:26:14.247 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 456 k 2024-09-12T06:26:14.247 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-09-12T06:26:14.437 INFO:teuthology.orchestra.run.smithi152.stdout:librados-devel-19.3.0-4883.gd61b0886.el9.x86_64 669 kB/s | 127 kB 00:00 2024-09-12T06:26:14.438 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-09-12T06:26:14.438 INFO:teuthology.orchestra.run.smithi152.stdout:Total 662 kB/s | 127 kB 00:00 2024-09-12T06:26:14.438 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-09-12T06:26:14.444 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-09-12T06:26:14.445 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-09-12T06:26:14.512 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-09-12T06:26:14.513 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-09-12T06:26:14.668 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-09-12T06:26:14.785 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : librados-devel-2:19.3.0-4883.gd61b0886.el9.x86_64 1/1 2024-09-12T06:26:15.241 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librados-devel-2:19.3.0-4883.gd61b0886.el9.x86_64 1/1 2024-09-12T06:26:15.517 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librados-devel-2:19.3.0-4883.gd61b0886.el9.x86_64 1/1 2024-09-12T06:26:15.517 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:26:15.517 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-09-12T06:26:15.517 INFO:teuthology.orchestra.run.smithi152.stdout: librados-devel-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T06:26:15.517 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:26:15.518 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-09-12T06:26:15.763 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install libcephfs2 2024-09-12T06:26:16.025 INFO:teuthology.orchestra.run.smithi096.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 861 kB/s | 16 MB 00:19 2024-09-12T06:26:16.026 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-09-12T06:26:16.026 INFO:teuthology.orchestra.run.smithi096.stdout:Total 1.8 MB/s | 38 MB 00:21 2024-09-12T06:26:16.347 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:45 ago on Thu 12 Sep 2024 06:24:31 AM UTC. 2024-09-12T06:26:16.383 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-09-12T06:26:16.410 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-09-12T06:26:16.410 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-09-12T06:26:16.439 INFO:teuthology.orchestra.run.smithi152.stdout:Package libcephfs2-2:19.3.0-4883.gd61b0886.el9.x86_64 is already installed. 2024-09-12T06:26:16.483 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-09-12T06:26:16.484 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-09-12T06:26:16.484 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-09-12T06:26:16.556 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install libcephfs-devel 2024-09-12T06:26:16.839 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-09-12T06:26:16.839 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-09-12T06:26:17.125 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:46 ago on Thu 12 Sep 2024 06:24:31 AM UTC. 2024-09-12T06:26:17.260 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-09-12T06:26:17.261 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-12T06:26:17.261 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repo Size 2024-09-12T06:26:17.261 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-12T06:26:17.262 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-09-12T06:26:17.262 INFO:teuthology.orchestra.run.smithi152.stdout: libcephfs-devel x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 32 k 2024-09-12T06:26:17.262 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:26:17.262 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-09-12T06:26:17.262 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-12T06:26:17.262 INFO:teuthology.orchestra.run.smithi152.stdout:Install 1 Package 2024-09-12T06:26:17.262 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:26:17.262 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 32 k 2024-09-12T06:26:17.262 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 142 k 2024-09-12T06:26:17.262 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-09-12T06:26:17.423 INFO:teuthology.orchestra.run.smithi152.stdout:libcephfs-devel-19.3.0-4883.gd61b0886.el9.x86_6 197 kB/s | 32 kB 00:00 2024-09-12T06:26:17.424 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-09-12T06:26:17.424 INFO:teuthology.orchestra.run.smithi152.stdout:Total 195 kB/s | 32 kB 00:00 2024-09-12T06:26:17.424 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-09-12T06:26:17.428 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-09-12T06:26:17.428 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-09-12T06:26:17.449 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-09-12T06:26:17.449 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-09-12T06:26:17.542 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-09-12T06:26:17.603 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-09-12T06:26:17.646 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libcephfs-devel-2:19.3.0-4883.gd61b0886.el9.x86_64 1/1 2024-09-12T06:26:17.684 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-12T06:26:17.745 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-12T06:26:17.855 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-12T06:26:17.899 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-12T06:26:18.037 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4883.gd61b0886.el9.x86_64 1/1 2024-09-12T06:26:18.083 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-12T06:26:18.114 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-12T06:26:18.228 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-12T06:26:18.264 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libcephfs-devel-2:19.3.0-4883.gd61b0886.el9.x86_64 1/1 2024-09-12T06:26:18.264 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:26:18.264 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-09-12T06:26:18.264 INFO:teuthology.orchestra.run.smithi152.stdout: libcephfs-devel-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T06:26:18.264 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:26:18.265 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-09-12T06:26:18.473 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install librados2 2024-09-12T06:26:18.882 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-12T06:26:19.003 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-12T06:26:19.035 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:48 ago on Thu 12 Sep 2024 06:24:31 AM UTC. 2024-09-12T06:26:19.125 INFO:teuthology.orchestra.run.smithi152.stdout:Package librados2-2:19.3.0-4883.gd61b0886.el9.x86_64 is already installed. 2024-09-12T06:26:19.168 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-09-12T06:26:19.169 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-09-12T06:26:19.169 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-09-12T06:26:19.228 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install librbd1 2024-09-12T06:26:19.794 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:48 ago on Thu 12 Sep 2024 06:24:31 AM UTC. 2024-09-12T06:26:19.884 INFO:teuthology.orchestra.run.smithi152.stdout:Package librbd1-2:19.3.0-4883.gd61b0886.el9.x86_64 is already installed. 2024-09-12T06:26:19.929 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-09-12T06:26:19.930 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-09-12T06:26:19.930 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-09-12T06:26:19.997 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install python3-rados 2024-09-12T06:26:20.230 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-12T06:26:20.569 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:49 ago on Thu 12 Sep 2024 06:24:31 AM UTC. 2024-09-12T06:26:20.661 INFO:teuthology.orchestra.run.smithi152.stdout:Package python3-rados-2:19.3.0-4883.gd61b0886.el9.x86_64 is already installed. 2024-09-12T06:26:20.705 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-09-12T06:26:20.706 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-09-12T06:26:20.706 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-09-12T06:26:20.772 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install python3-rgw 2024-09-12T06:26:21.253 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-12T06:26:21.281 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4883.gd61b0 12/12 2024-09-12T06:26:21.338 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:50 ago on Thu 12 Sep 2024 06:24:31 AM UTC. 2024-09-12T06:26:21.430 INFO:teuthology.orchestra.run.smithi152.stdout:Package python3-rgw-2:19.3.0-4883.gd61b0886.el9.x86_64 is already installed. 2024-09-12T06:26:21.473 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-09-12T06:26:21.474 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-09-12T06:26:21.475 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-09-12T06:26:21.537 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install python3-cephfs 2024-09-12T06:26:22.101 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:51 ago on Thu 12 Sep 2024 06:24:31 AM UTC. 2024-09-12T06:26:22.191 INFO:teuthology.orchestra.run.smithi152.stdout:Package python3-cephfs-2:19.3.0-4883.gd61b0886.el9.x86_64 is already installed. 2024-09-12T06:26:22.234 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-09-12T06:26:22.235 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-09-12T06:26:22.235 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-09-12T06:26:22.294 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install python3-rbd 2024-09-12T06:26:22.865 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:51 ago on Thu 12 Sep 2024 06:24:31 AM UTC. 2024-09-12T06:26:22.956 INFO:teuthology.orchestra.run.smithi152.stdout:Package python3-rbd-2:19.3.0-4883.gd61b0886.el9.x86_64 is already installed. 2024-09-12T06:26:22.999 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-09-12T06:26:23.000 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-09-12T06:26:23.000 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-09-12T06:26:23.063 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install rbd-fuse 2024-09-12T06:26:23.623 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:52 ago on Thu 12 Sep 2024 06:24:31 AM UTC. 2024-09-12T06:26:23.756 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-09-12T06:26:23.757 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-12T06:26:23.757 INFO:teuthology.orchestra.run.smithi152.stdout: Package Architecture Version Repository Size 2024-09-12T06:26:23.757 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-12T06:26:23.757 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-09-12T06:26:23.757 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-fuse x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 85 k 2024-09-12T06:26:23.757 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:26:23.757 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-09-12T06:26:23.757 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-12T06:26:23.757 INFO:teuthology.orchestra.run.smithi152.stdout:Install 1 Package 2024-09-12T06:26:23.757 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:26:23.758 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 85 k 2024-09-12T06:26:23.758 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 231 k 2024-09-12T06:26:23.758 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-09-12T06:26:23.901 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4883.gd61b0 12/12 2024-09-12T06:26:23.901 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4883.gd61b0 1/12 2024-09-12T06:26:23.902 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-12T06:26:23.902 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-12T06:26:23.902 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-12T06:26:23.902 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-12T06:26:23.902 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-12T06:26:23.902 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-12T06:26:23.902 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-12T06:26:23.902 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-12T06:26:23.902 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-12T06:26:23.902 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-12T06:26:23.951 INFO:teuthology.orchestra.run.smithi152.stdout:rbd-fuse-19.3.0-4883.gd61b0886.el9.x86_64.rpm 440 kB/s | 85 kB 00:00 2024-09-12T06:26:23.952 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-09-12T06:26:23.952 INFO:teuthology.orchestra.run.smithi152.stdout:Total 437 kB/s | 85 kB 00:00 2024-09-12T06:26:23.952 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-09-12T06:26:23.959 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-09-12T06:26:23.959 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-09-12T06:26:24.022 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-09-12T06:26:24.022 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-09-12T06:26:24.171 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-09-12T06:26:24.273 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-12T06:26:24.273 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:26:24.273 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-09-12T06:26:24.273 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4883.gd61b0886.el9.noarch 2024-09-12T06:26:24.273 INFO:teuthology.orchestra.run.smithi096.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-12T06:26:24.273 INFO:teuthology.orchestra.run.smithi096.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-12T06:26:24.273 INFO:teuthology.orchestra.run.smithi096.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-12T06:26:24.274 INFO:teuthology.orchestra.run.smithi096.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-12T06:26:24.274 INFO:teuthology.orchestra.run.smithi096.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-12T06:26:24.274 INFO:teuthology.orchestra.run.smithi096.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-12T06:26:24.274 INFO:teuthology.orchestra.run.smithi096.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-12T06:26:24.274 INFO:teuthology.orchestra.run.smithi096.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-12T06:26:24.274 INFO:teuthology.orchestra.run.smithi096.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-12T06:26:24.274 INFO:teuthology.orchestra.run.smithi096.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-12T06:26:24.274 INFO:teuthology.orchestra.run.smithi096.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-12T06:26:24.274 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:26:24.274 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-12T06:26:24.274 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : rbd-fuse-2:19.3.0-4883.gd61b0886.el9.x86_64 1/1 2024-09-12T06:26:24.655 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4883.gd61b0886.el9.x86_64 1/1 2024-09-12T06:26:24.692 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-mgr-rook 2024-09-12T06:26:24.912 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : rbd-fuse-2:19.3.0-4883.gd61b0886.el9.x86_64 1/1 2024-09-12T06:26:24.912 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:26:24.912 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-09-12T06:26:24.912 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-fuse-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T06:26:24.912 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:26:24.912 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-09-12T06:26:25.194 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install rbd-mirror 2024-09-12T06:26:25.268 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:41 ago on Thu 12 Sep 2024 06:24:44 AM UTC. 2024-09-12T06:26:25.400 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-12T06:26:25.401 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-12T06:26:25.402 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repository Size 2024-09-12T06:26:25.402 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-12T06:26:25.402 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-09-12T06:26:25.402 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-rook noarch 2:19.3.0-4883.gd61b0886.el9 ceph-noarch 49 k 2024-09-12T06:26:25.402 INFO:teuthology.orchestra.run.smithi096.stdout:Installing dependencies: 2024-09-12T06:26:25.402 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-12T06:26:25.402 INFO:teuthology.orchestra.run.smithi096.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-12T06:26:25.402 INFO:teuthology.orchestra.run.smithi096.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-12T06:26:25.402 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-12T06:26:25.402 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-12T06:26:25.402 INFO:teuthology.orchestra.run.smithi096.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-12T06:26:25.402 INFO:teuthology.orchestra.run.smithi096.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-12T06:26:25.403 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-12T06:26:25.403 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-12T06:26:25.403 INFO:teuthology.orchestra.run.smithi096.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-12T06:26:25.403 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-12T06:26:25.403 INFO:teuthology.orchestra.run.smithi096.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-12T06:26:25.403 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:26:25.403 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-09-12T06:26:25.403 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-12T06:26:25.403 INFO:teuthology.orchestra.run.smithi096.stdout:Install 13 Packages 2024-09-12T06:26:25.403 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:26:25.404 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 2.2 M 2024-09-12T06:26:25.404 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 26 M 2024-09-12T06:26:25.404 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-09-12T06:26:25.691 INFO:teuthology.orchestra.run.smithi096.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 324 kB/s | 26 kB 00:00 2024-09-12T06:26:25.716 INFO:teuthology.orchestra.run.smithi096.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 184 kB/s | 19 kB 00:00 2024-09-12T06:26:25.775 INFO:teuthology.orchestra.run.smithi096.stdout:(3/13): ceph-mgr-rook-19.3.0-4883.gd61b0886.el9 302 kB/s | 49 kB 00:00 2024-09-12T06:26:25.788 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:54 ago on Thu 12 Sep 2024 06:24:31 AM UTC. 2024-09-12T06:26:25.800 INFO:teuthology.orchestra.run.smithi096.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.0 MB/s | 222 kB 00:00 2024-09-12T06:26:25.826 INFO:teuthology.orchestra.run.smithi096.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 159 kB 00:00 2024-09-12T06:26:25.859 INFO:teuthology.orchestra.run.smithi096.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 915 kB/s | 54 kB 00:00 2024-09-12T06:26:25.927 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-09-12T06:26:25.928 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-12T06:26:25.928 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repo Size 2024-09-12T06:26:25.928 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-12T06:26:25.928 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-09-12T06:26:25.928 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-mirror x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 3.2 M 2024-09-12T06:26:25.928 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:26:25.928 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-09-12T06:26:25.928 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-12T06:26:25.929 INFO:teuthology.orchestra.run.smithi152.stdout:Install 1 Package 2024-09-12T06:26:25.929 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:26:25.929 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 3.2 M 2024-09-12T06:26:25.929 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 14 M 2024-09-12T06:26:25.929 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-09-12T06:26:25.934 INFO:teuthology.orchestra.run.smithi096.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.7 MB/s | 279 kB 00:00 2024-09-12T06:26:26.043 INFO:teuthology.orchestra.run.smithi096.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 77 kB/s | 14 kB 00:00 2024-09-12T06:26:26.060 INFO:teuthology.orchestra.run.smithi096.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 137 kB/s | 32 kB 00:00 2024-09-12T06:26:26.193 INFO:teuthology.orchestra.run.smithi096.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 443 kB/s | 59 kB 00:00 2024-09-12T06:26:26.254 INFO:teuthology.orchestra.run.smithi152.stdout:rbd-mirror-19.3.0-4883.gd61b0886.el9.x86_64.rpm 9.8 MB/s | 3.2 MB 00:00 2024-09-12T06:26:26.255 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-09-12T06:26:26.255 INFO:teuthology.orchestra.run.smithi152.stdout:Total 9.8 MB/s | 3.2 MB 00:00 2024-09-12T06:26:26.255 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-09-12T06:26:26.265 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-09-12T06:26:26.265 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-09-12T06:26:26.302 INFO:teuthology.orchestra.run.smithi096.stdout:(11/13): python3-google-auth-2.34.0-1.el9.noarc 599 kB/s | 220 kB 00:00 2024-09-12T06:26:26.327 INFO:teuthology.orchestra.run.smithi096.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 671 kB/s | 90 kB 00:00 2024-09-12T06:26:26.347 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-09-12T06:26:26.348 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-09-12T06:26:26.369 INFO:teuthology.orchestra.run.smithi096.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.1 MB/s | 1.0 MB 00:00 2024-09-12T06:26:26.372 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-09-12T06:26:26.372 INFO:teuthology.orchestra.run.smithi096.stdout:Total 2.3 MB/s | 2.2 MB 00:00 2024-09-12T06:26:26.436 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-09-12T06:26:26.451 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-09-12T06:26:26.451 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-09-12T06:26:26.539 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-09-12T06:26:26.541 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-09-12T06:26:26.793 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-09-12T06:26:26.855 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : rbd-mirror-2:19.3.0-4883.gd61b0886.el9.x86_64 1/1 2024-09-12T06:26:26.880 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4883.gd61b0886.el9.x86_64 1/1 2024-09-12T06:26:26.880 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-12T06:26:26.881 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-12T06:26:26.881 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-12T06:26:26.881 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-12T06:26:26.881 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:26:27.042 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-09-12T06:26:27.215 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-12T06:26:27.288 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-12T06:26:27.350 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-12T06:26:27.397 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-12T06:26:27.438 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-12T06:26:27.547 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-12T06:26:27.649 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-12T06:26:27.698 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-12T06:26:28.023 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : rbd-mirror-2:19.3.0-4883.gd61b0886.el9.x86_64 1/1 2024-09-12T06:26:28.023 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:26:28.023 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-09-12T06:26:28.023 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-mirror-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T06:26:28.023 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:26:28.023 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-09-12T06:26:28.274 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install rbd-nbd 2024-09-12T06:26:28.418 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-12T06:26:28.463 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-12T06:26:28.496 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-12T06:26:28.544 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-12T06:26:28.566 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mgr-rook-2:19.3.0-4883.gd61b0886.el9.noarch 13/13 2024-09-12T06:26:28.854 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:57 ago on Thu 12 Sep 2024 06:24:31 AM UTC. 2024-09-12T06:26:28.993 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-09-12T06:26:28.993 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-12T06:26:28.993 INFO:teuthology.orchestra.run.smithi152.stdout: Package Architecture Version Repository Size 2024-09-12T06:26:28.994 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-12T06:26:28.994 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-09-12T06:26:28.994 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-nbd x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 171 k 2024-09-12T06:26:28.994 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:26:28.994 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-09-12T06:26:28.994 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-09-12T06:26:28.994 INFO:teuthology.orchestra.run.smithi152.stdout:Install 1 Package 2024-09-12T06:26:28.994 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:26:28.994 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 171 k 2024-09-12T06:26:28.994 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 490 k 2024-09-12T06:26:28.994 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-09-12T06:26:29.208 INFO:teuthology.orchestra.run.smithi152.stdout:rbd-nbd-19.3.0-4883.gd61b0886.el9.x86_64.rpm 801 kB/s | 171 kB 00:00 2024-09-12T06:26:29.208 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-09-12T06:26:29.209 INFO:teuthology.orchestra.run.smithi152.stdout:Total 795 kB/s | 171 kB 00:00 2024-09-12T06:26:29.209 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-09-12T06:26:29.216 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-09-12T06:26:29.217 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-09-12T06:26:29.280 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-09-12T06:26:29.280 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-09-12T06:26:29.396 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4883.gd61b0886.el9.noarch 13/13 2024-09-12T06:26:29.396 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4883.gd61b0886.el9.noarch 1/13 2024-09-12T06:26:29.397 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-12T06:26:29.397 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-12T06:26:29.397 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-12T06:26:29.397 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-12T06:26:29.397 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-12T06:26:29.397 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-12T06:26:29.397 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-12T06:26:29.397 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-12T06:26:29.397 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-12T06:26:29.397 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-12T06:26:29.397 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-12T06:26:29.429 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-09-12T06:26:29.522 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : rbd-nbd-2:19.3.0-4883.gd61b0886.el9.x86_64 1/1 2024-09-12T06:26:29.727 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-12T06:26:29.728 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:26:29.728 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-09-12T06:26:29.728 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-rook-2:19.3.0-4883.gd61b0886.el9.noarch 2024-09-12T06:26:29.728 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-12T06:26:29.728 INFO:teuthology.orchestra.run.smithi096.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-12T06:26:29.728 INFO:teuthology.orchestra.run.smithi096.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-12T06:26:29.728 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-12T06:26:29.728 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-12T06:26:29.728 INFO:teuthology.orchestra.run.smithi096.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-12T06:26:29.728 INFO:teuthology.orchestra.run.smithi096.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-12T06:26:29.728 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-12T06:26:29.729 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-12T06:26:29.729 INFO:teuthology.orchestra.run.smithi096.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-12T06:26:29.729 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-12T06:26:29.729 INFO:teuthology.orchestra.run.smithi096.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-12T06:26:29.729 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:26:29.729 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-12T06:26:29.948 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4883.gd61b0886.el9.x86_64 1/1 2024-09-12T06:26:30.002 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-mgr-cephadm 2024-09-12T06:26:30.198 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : rbd-nbd-2:19.3.0-4883.gd61b0886.el9.x86_64 1/1 2024-09-12T06:26:30.198 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:26:30.198 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-09-12T06:26:30.198 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-nbd-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T06:26:30.199 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:26:30.199 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-09-12T06:26:30.434 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install rbd-mirror 2024-09-12T06:26:30.567 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:46 ago on Thu 12 Sep 2024 06:24:44 AM UTC. 2024-09-12T06:26:30.699 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-12T06:26:30.700 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-12T06:26:30.700 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repository Size 2024-09-12T06:26:30.700 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-12T06:26:30.700 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-09-12T06:26:30.700 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4883.gd61b0886.el9 ceph-noarch 154 k 2024-09-12T06:26:30.701 INFO:teuthology.orchestra.run.smithi096.stdout:Installing dependencies: 2024-09-12T06:26:30.701 INFO:teuthology.orchestra.run.smithi096.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-12T06:26:30.701 INFO:teuthology.orchestra.run.smithi096.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-12T06:26:30.701 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-12T06:26:30.701 INFO:teuthology.orchestra.run.smithi096.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-12T06:26:30.701 INFO:teuthology.orchestra.run.smithi096.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-12T06:26:30.701 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:26:30.701 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-09-12T06:26:30.701 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-12T06:26:30.701 INFO:teuthology.orchestra.run.smithi096.stdout:Install 6 Packages 2024-09-12T06:26:30.701 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:26:30.702 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 7.0 M 2024-09-12T06:26:30.702 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 33 M 2024-09-12T06:26:30.702 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-09-12T06:26:30.981 INFO:teuthology.orchestra.run.smithi096.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.6 MB/s | 248 kB 00:00 2024-09-12T06:26:31.014 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:02:00 ago on Thu 12 Sep 2024 06:24:31 AM UTC. 2024-09-12T06:26:31.048 INFO:teuthology.orchestra.run.smithi096.stdout:(2/6): ceph-mgr-cephadm-19.3.0-4883.gd61b0886.e 713 kB/s | 154 kB 00:00 2024-09-12T06:26:31.105 INFO:teuthology.orchestra.run.smithi152.stdout:Package rbd-mirror-2:19.3.0-4883.gd61b0886.el9.x86_64 is already installed. 2024-09-12T06:26:31.150 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-09-12T06:26:31.152 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-09-12T06:26:31.152 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-09-12T06:26:31.156 INFO:teuthology.orchestra.run.smithi096.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 18 MB/s | 6.0 MB 00:00 2024-09-12T06:26:31.215 DEBUG:teuthology.parallel:result is None 2024-09-12T06:26:31.232 INFO:teuthology.orchestra.run.smithi096.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 2.1 MB/s | 548 kB 00:00 2024-09-12T06:26:31.257 INFO:teuthology.orchestra.run.smithi096.stdout:(5/6): python3-natsort-7.1.1-5.el9.noarch.rpm 276 kB/s | 58 kB 00:00 2024-09-12T06:26:31.282 INFO:teuthology.orchestra.run.smithi096.stdout:(6/6): python3-typing-extensions-4.12.2-2.el9.n 600 kB/s | 74 kB 00:00 2024-09-12T06:26:31.285 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-09-12T06:26:31.286 INFO:teuthology.orchestra.run.smithi096.stdout:Total 12 MB/s | 7.0 MB 00:00 2024-09-12T06:26:31.357 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-09-12T06:26:31.368 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-09-12T06:26:31.368 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-09-12T06:26:31.467 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-09-12T06:26:31.468 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-09-12T06:26:31.773 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-09-12T06:26:31.949 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-12T06:26:32.001 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-12T06:26:32.525 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-12T06:26:32.621 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-12T06:26:32.688 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-12T06:26:32.713 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4883.gd61b0886.el9.noarch 6/6 2024-09-12T06:26:33.476 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4883.gd61b0886.el9.noarch 6/6 2024-09-12T06:26:33.476 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4883.gd61b0886.el9.noarch 1/6 2024-09-12T06:26:33.476 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-12T06:26:33.476 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-12T06:26:33.476 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-12T06:26:33.476 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-12T06:26:33.759 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-12T06:26:33.759 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:26:33.759 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-09-12T06:26:33.759 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-cephadm-2:19.3.0-4883.gd61b0886.el9.noarch 2024-09-12T06:26:33.759 INFO:teuthology.orchestra.run.smithi096.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-12T06:26:33.760 INFO:teuthology.orchestra.run.smithi096.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-12T06:26:33.760 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-12T06:26:33.760 INFO:teuthology.orchestra.run.smithi096.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-12T06:26:33.760 INFO:teuthology.orchestra.run.smithi096.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-12T06:26:33.760 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:26:33.760 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-12T06:26:34.012 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-fuse 2024-09-12T06:26:34.580 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:50 ago on Thu 12 Sep 2024 06:24:44 AM UTC. 2024-09-12T06:26:34.713 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-12T06:26:34.714 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-12T06:26:34.715 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repository Size 2024-09-12T06:26:34.715 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-12T06:26:34.715 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-09-12T06:26:34.715 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-fuse x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 900 k 2024-09-12T06:26:34.715 INFO:teuthology.orchestra.run.smithi096.stdout:Installing dependencies: 2024-09-12T06:26:34.715 INFO:teuthology.orchestra.run.smithi096.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-12T06:26:34.715 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:26:34.715 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-09-12T06:26:34.715 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-12T06:26:34.715 INFO:teuthology.orchestra.run.smithi096.stdout:Install 2 Packages 2024-09-12T06:26:34.715 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:26:34.715 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 980 k 2024-09-12T06:26:34.715 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 2.9 M 2024-09-12T06:26:34.716 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-09-12T06:26:34.993 INFO:teuthology.orchestra.run.smithi096.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 517 kB/s | 80 kB 00:00 2024-09-12T06:26:35.126 INFO:teuthology.orchestra.run.smithi096.stdout:(2/2): ceph-fuse-19.3.0-4883.gd61b0886.el9.x86_ 3.1 MB/s | 900 kB 00:00 2024-09-12T06:26:35.127 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-09-12T06:26:35.127 INFO:teuthology.orchestra.run.smithi096.stdout:Total 2.3 MB/s | 980 kB 00:00 2024-09-12T06:26:35.138 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-09-12T06:26:35.160 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-09-12T06:26:35.160 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-09-12T06:26:35.241 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-09-12T06:26:35.242 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-09-12T06:26:35.432 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-09-12T06:26:35.570 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-12T06:26:35.642 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-fuse-2:19.3.0-4883.gd61b0886.el9.x86_64 2/2 2024-09-12T06:26:36.379 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4883.gd61b0886.el9.x86_64 2/2 2024-09-12T06:26:36.379 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-fuse-2:19.3.0-4883.gd61b0886.el9.x86_64 1/2 2024-09-12T06:26:36.658 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-12T06:26:36.659 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:26:36.659 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-09-12T06:26:36.659 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-fuse-2:19.3.0-4883.gd61b0886.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-12T06:26:36.659 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:26:36.659 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-12T06:26:36.947 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-volume 2024-09-12T06:26:37.526 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:53 ago on Thu 12 Sep 2024 06:24:44 AM UTC. 2024-09-12T06:26:37.665 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-12T06:26:37.666 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-12T06:26:37.666 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repository Size 2024-09-12T06:26:37.666 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-12T06:26:37.666 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-09-12T06:26:37.666 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-volume noarch 2:19.3.0-4883.gd61b0886.el9 ceph-noarch 285 k 2024-09-12T06:26:37.666 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:26:37.666 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-09-12T06:26:37.666 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-12T06:26:37.666 INFO:teuthology.orchestra.run.smithi096.stdout:Install 1 Package 2024-09-12T06:26:37.666 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:26:37.666 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 285 k 2024-09-12T06:26:37.667 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 1.3 M 2024-09-12T06:26:37.667 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-09-12T06:26:37.890 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-volume-19.3.0-4883.gd61b0886.el9.noarch.rp 1.2 MB/s | 285 kB 00:00 2024-09-12T06:26:37.891 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-09-12T06:26:37.891 INFO:teuthology.orchestra.run.smithi096.stdout:Total 1.2 MB/s | 285 kB 00:00 2024-09-12T06:26:37.891 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-09-12T06:26:37.899 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-09-12T06:26:37.900 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-09-12T06:26:37.933 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-09-12T06:26:37.933 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-09-12T06:26:38.151 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-09-12T06:26:38.234 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-volume-2:19.3.0-4883.gd61b0886.el9.noarch 1/1 2024-09-12T06:26:38.250 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-volume-2:19.3.0-4883.gd61b0886.el9.noarch 1/1 2024-09-12T06:26:38.250 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-12T06:26:38.250 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-12T06:26:38.250 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:26:39.301 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-volume-2:19.3.0-4883.gd61b0886.el9.noarch 1/1 2024-09-12T06:26:39.302 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:26:39.302 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-09-12T06:26:39.302 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-volume-2:19.3.0-4883.gd61b0886.el9.noarch 2024-09-12T06:26:39.302 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:26:39.302 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-12T06:26:39.636 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install librados-devel 2024-09-12T06:26:40.213 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:56 ago on Thu 12 Sep 2024 06:24:44 AM UTC. 2024-09-12T06:26:40.352 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-12T06:26:40.353 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-12T06:26:40.353 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repo Size 2024-09-12T06:26:40.353 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-12T06:26:40.353 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-09-12T06:26:40.354 INFO:teuthology.orchestra.run.smithi096.stdout: librados-devel x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 127 k 2024-09-12T06:26:40.354 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:26:40.354 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-09-12T06:26:40.354 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-12T06:26:40.354 INFO:teuthology.orchestra.run.smithi096.stdout:Install 1 Package 2024-09-12T06:26:40.354 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:26:40.354 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 127 k 2024-09-12T06:26:40.354 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 456 k 2024-09-12T06:26:40.354 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-09-12T06:26:40.571 INFO:teuthology.orchestra.run.smithi096.stdout:librados-devel-19.3.0-4883.gd61b0886.el9.x86_64 585 kB/s | 127 kB 00:00 2024-09-12T06:26:40.571 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-09-12T06:26:40.572 INFO:teuthology.orchestra.run.smithi096.stdout:Total 581 kB/s | 127 kB 00:00 2024-09-12T06:26:40.572 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-09-12T06:26:40.577 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-09-12T06:26:40.577 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-09-12T06:26:40.641 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-09-12T06:26:40.641 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-09-12T06:26:40.791 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-09-12T06:26:40.902 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : librados-devel-2:19.3.0-4883.gd61b0886.el9.x86_64 1/1 2024-09-12T06:26:41.361 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: librados-devel-2:19.3.0-4883.gd61b0886.el9.x86_64 1/1 2024-09-12T06:26:41.641 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librados-devel-2:19.3.0-4883.gd61b0886.el9.x86_64 1/1 2024-09-12T06:26:41.642 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:26:41.642 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-09-12T06:26:41.642 INFO:teuthology.orchestra.run.smithi096.stdout: librados-devel-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T06:26:41.642 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:26:41.642 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-12T06:26:41.878 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install libcephfs2 2024-09-12T06:26:42.464 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:58 ago on Thu 12 Sep 2024 06:24:44 AM UTC. 2024-09-12T06:26:42.559 INFO:teuthology.orchestra.run.smithi096.stdout:Package libcephfs2-2:19.3.0-4883.gd61b0886.el9.x86_64 is already installed. 2024-09-12T06:26:42.603 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-12T06:26:42.605 INFO:teuthology.orchestra.run.smithi096.stdout:Nothing to do. 2024-09-12T06:26:42.605 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-12T06:26:42.675 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install libcephfs-devel 2024-09-12T06:26:43.243 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:59 ago on Thu 12 Sep 2024 06:24:44 AM UTC. 2024-09-12T06:26:43.376 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-12T06:26:43.377 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-12T06:26:43.377 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repo Size 2024-09-12T06:26:43.377 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-12T06:26:43.377 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-09-12T06:26:43.377 INFO:teuthology.orchestra.run.smithi096.stdout: libcephfs-devel x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 32 k 2024-09-12T06:26:43.377 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:26:43.377 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-09-12T06:26:43.378 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-12T06:26:43.378 INFO:teuthology.orchestra.run.smithi096.stdout:Install 1 Package 2024-09-12T06:26:43.378 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:26:43.378 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 32 k 2024-09-12T06:26:43.378 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 142 k 2024-09-12T06:26:43.378 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-09-12T06:26:43.524 INFO:teuthology.orchestra.run.smithi096.stdout:libcephfs-devel-19.3.0-4883.gd61b0886.el9.x86_6 217 kB/s | 32 kB 00:00 2024-09-12T06:26:43.525 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-09-12T06:26:43.525 INFO:teuthology.orchestra.run.smithi096.stdout:Total 215 kB/s | 32 kB 00:00 2024-09-12T06:26:43.525 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-09-12T06:26:43.529 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-09-12T06:26:43.529 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-09-12T06:26:43.545 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-09-12T06:26:43.545 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-09-12T06:26:43.637 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-09-12T06:26:43.738 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libcephfs-devel-2:19.3.0-4883.gd61b0886.el9.x86_64 1/1 2024-09-12T06:26:44.176 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4883.gd61b0886.el9.x86_64 1/1 2024-09-12T06:26:44.424 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libcephfs-devel-2:19.3.0-4883.gd61b0886.el9.x86_64 1/1 2024-09-12T06:26:44.424 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:26:44.424 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-09-12T06:26:44.424 INFO:teuthology.orchestra.run.smithi096.stdout: libcephfs-devel-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T06:26:44.424 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:26:44.424 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-12T06:26:44.630 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install librados2 2024-09-12T06:26:45.190 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:02:01 ago on Thu 12 Sep 2024 06:24:44 AM UTC. 2024-09-12T06:26:45.280 INFO:teuthology.orchestra.run.smithi096.stdout:Package librados2-2:19.3.0-4883.gd61b0886.el9.x86_64 is already installed. 2024-09-12T06:26:45.323 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-12T06:26:45.324 INFO:teuthology.orchestra.run.smithi096.stdout:Nothing to do. 2024-09-12T06:26:45.324 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-12T06:26:45.399 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install librbd1 2024-09-12T06:26:45.956 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:02:01 ago on Thu 12 Sep 2024 06:24:44 AM UTC. 2024-09-12T06:26:46.044 INFO:teuthology.orchestra.run.smithi096.stdout:Package librbd1-2:19.3.0-4883.gd61b0886.el9.x86_64 is already installed. 2024-09-12T06:26:46.087 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-12T06:26:46.088 INFO:teuthology.orchestra.run.smithi096.stdout:Nothing to do. 2024-09-12T06:26:46.088 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-12T06:26:46.164 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install python3-rados 2024-09-12T06:26:46.732 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:02:02 ago on Thu 12 Sep 2024 06:24:44 AM UTC. 2024-09-12T06:26:46.820 INFO:teuthology.orchestra.run.smithi096.stdout:Package python3-rados-2:19.3.0-4883.gd61b0886.el9.x86_64 is already installed. 2024-09-12T06:26:46.862 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-12T06:26:46.863 INFO:teuthology.orchestra.run.smithi096.stdout:Nothing to do. 2024-09-12T06:26:46.863 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-12T06:26:46.937 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install python3-rgw 2024-09-12T06:26:47.508 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:02:03 ago on Thu 12 Sep 2024 06:24:44 AM UTC. 2024-09-12T06:26:47.598 INFO:teuthology.orchestra.run.smithi096.stdout:Package python3-rgw-2:19.3.0-4883.gd61b0886.el9.x86_64 is already installed. 2024-09-12T06:26:47.641 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-12T06:26:47.642 INFO:teuthology.orchestra.run.smithi096.stdout:Nothing to do. 2024-09-12T06:26:47.642 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-12T06:26:47.718 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install python3-cephfs 2024-09-12T06:26:48.278 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:02:04 ago on Thu 12 Sep 2024 06:24:44 AM UTC. 2024-09-12T06:26:48.368 INFO:teuthology.orchestra.run.smithi096.stdout:Package python3-cephfs-2:19.3.0-4883.gd61b0886.el9.x86_64 is already installed. 2024-09-12T06:26:48.411 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-12T06:26:48.412 INFO:teuthology.orchestra.run.smithi096.stdout:Nothing to do. 2024-09-12T06:26:48.412 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-12T06:26:48.484 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install python3-rbd 2024-09-12T06:26:49.052 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:02:05 ago on Thu 12 Sep 2024 06:24:44 AM UTC. 2024-09-12T06:26:49.142 INFO:teuthology.orchestra.run.smithi096.stdout:Package python3-rbd-2:19.3.0-4883.gd61b0886.el9.x86_64 is already installed. 2024-09-12T06:26:49.186 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-12T06:26:49.186 INFO:teuthology.orchestra.run.smithi096.stdout:Nothing to do. 2024-09-12T06:26:49.186 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-12T06:26:49.259 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install rbd-fuse 2024-09-12T06:26:49.823 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:02:05 ago on Thu 12 Sep 2024 06:24:44 AM UTC. 2024-09-12T06:26:49.956 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-12T06:26:49.957 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-12T06:26:49.957 INFO:teuthology.orchestra.run.smithi096.stdout: Package Architecture Version Repository Size 2024-09-12T06:26:49.957 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-12T06:26:49.957 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-09-12T06:26:49.957 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-fuse x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 85 k 2024-09-12T06:26:49.958 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:26:49.958 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-09-12T06:26:49.958 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-12T06:26:49.958 INFO:teuthology.orchestra.run.smithi096.stdout:Install 1 Package 2024-09-12T06:26:49.958 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:26:49.958 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 85 k 2024-09-12T06:26:49.958 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 231 k 2024-09-12T06:26:49.958 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-09-12T06:26:50.153 INFO:teuthology.orchestra.run.smithi096.stdout:rbd-fuse-19.3.0-4883.gd61b0886.el9.x86_64.rpm 437 kB/s | 85 kB 00:00 2024-09-12T06:26:50.153 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-09-12T06:26:50.154 INFO:teuthology.orchestra.run.smithi096.stdout:Total 434 kB/s | 85 kB 00:00 2024-09-12T06:26:50.154 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-09-12T06:26:50.160 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-09-12T06:26:50.160 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-09-12T06:26:50.224 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-09-12T06:26:50.225 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-09-12T06:26:50.374 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-09-12T06:26:50.476 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : rbd-fuse-2:19.3.0-4883.gd61b0886.el9.x86_64 1/1 2024-09-12T06:26:50.918 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4883.gd61b0886.el9.x86_64 1/1 2024-09-12T06:26:51.190 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : rbd-fuse-2:19.3.0-4883.gd61b0886.el9.x86_64 1/1 2024-09-12T06:26:51.190 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:26:51.190 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-09-12T06:26:51.190 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-fuse-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T06:26:51.190 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:26:51.190 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-12T06:26:51.439 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install rbd-mirror 2024-09-12T06:26:52.016 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:02:08 ago on Thu 12 Sep 2024 06:24:44 AM UTC. 2024-09-12T06:26:52.151 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-12T06:26:52.152 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-12T06:26:52.152 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repo Size 2024-09-12T06:26:52.152 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-12T06:26:52.152 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-09-12T06:26:52.152 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-mirror x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 3.2 M 2024-09-12T06:26:52.152 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:26:52.153 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-09-12T06:26:52.153 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-12T06:26:52.153 INFO:teuthology.orchestra.run.smithi096.stdout:Install 1 Package 2024-09-12T06:26:52.153 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:26:52.153 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 3.2 M 2024-09-12T06:26:52.153 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 14 M 2024-09-12T06:26:52.153 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-09-12T06:26:52.501 INFO:teuthology.orchestra.run.smithi096.stdout:rbd-mirror-19.3.0-4883.gd61b0886.el9.x86_64.rpm 9.2 MB/s | 3.2 MB 00:00 2024-09-12T06:26:52.501 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-09-12T06:26:52.502 INFO:teuthology.orchestra.run.smithi096.stdout:Total 9.1 MB/s | 3.2 MB 00:00 2024-09-12T06:26:52.502 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-09-12T06:26:52.511 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-09-12T06:26:52.511 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-09-12T06:26:52.592 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-09-12T06:26:52.592 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-09-12T06:26:53.034 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-09-12T06:26:53.108 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : rbd-mirror-2:19.3.0-4883.gd61b0886.el9.x86_64 1/1 2024-09-12T06:26:53.133 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4883.gd61b0886.el9.x86_64 1/1 2024-09-12T06:26:53.133 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-12T06:26:53.133 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-12T06:26:53.133 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-12T06:26:53.133 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-12T06:26:53.133 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:26:54.253 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : rbd-mirror-2:19.3.0-4883.gd61b0886.el9.x86_64 1/1 2024-09-12T06:26:54.254 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:26:54.254 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-09-12T06:26:54.254 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-mirror-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T06:26:54.254 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:26:54.254 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-12T06:26:54.524 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install rbd-nbd 2024-09-12T06:26:55.106 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:02:11 ago on Thu 12 Sep 2024 06:24:44 AM UTC. 2024-09-12T06:26:55.248 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-12T06:26:55.249 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-12T06:26:55.249 INFO:teuthology.orchestra.run.smithi096.stdout: Package Architecture Version Repository Size 2024-09-12T06:26:55.249 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-12T06:26:55.249 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-09-12T06:26:55.249 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-nbd x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 171 k 2024-09-12T06:26:55.249 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:26:55.249 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-09-12T06:26:55.249 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-12T06:26:55.250 INFO:teuthology.orchestra.run.smithi096.stdout:Install 1 Package 2024-09-12T06:26:55.250 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:26:55.250 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 171 k 2024-09-12T06:26:55.250 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 490 k 2024-09-12T06:26:55.250 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-09-12T06:26:55.462 INFO:teuthology.orchestra.run.smithi096.stdout:rbd-nbd-19.3.0-4883.gd61b0886.el9.x86_64.rpm 805 kB/s | 171 kB 00:00 2024-09-12T06:26:55.463 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-09-12T06:26:55.463 INFO:teuthology.orchestra.run.smithi096.stdout:Total 799 kB/s | 171 kB 00:00 2024-09-12T06:26:55.463 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-09-12T06:26:55.471 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-09-12T06:26:55.471 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-09-12T06:26:55.539 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-09-12T06:26:55.539 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-09-12T06:26:55.704 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-09-12T06:26:55.803 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : rbd-nbd-2:19.3.0-4883.gd61b0886.el9.x86_64 1/1 2024-09-12T06:26:56.269 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4883.gd61b0886.el9.x86_64 1/1 2024-09-12T06:26:56.530 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : rbd-nbd-2:19.3.0-4883.gd61b0886.el9.x86_64 1/1 2024-09-12T06:26:56.531 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:26:56.531 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-09-12T06:26:56.531 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-nbd-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T06:26:56.531 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:26:56.531 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-12T06:26:56.829 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install rbd-mirror 2024-09-12T06:26:57.413 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:02:13 ago on Thu 12 Sep 2024 06:24:44 AM UTC. 2024-09-12T06:26:57.507 INFO:teuthology.orchestra.run.smithi096.stdout:Package rbd-mirror-2:19.3.0-4883.gd61b0886.el9.x86_64 is already installed. 2024-09-12T06:26:57.551 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-12T06:26:57.552 INFO:teuthology.orchestra.run.smithi096.stdout:Nothing to do. 2024-09-12T06:26:57.552 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-12T06:26:57.625 DEBUG:teuthology.parallel:result is None 2024-09-12T06:26:57.626 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d61b08865d4d98738f45ff2e4806d7fb6d03eca1 2024-09-12T06:26:57.785 DEBUG:teuthology.orchestra.run.smithi096:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-12T06:26:57.806 INFO:teuthology.orchestra.run.smithi096.stdout:19.3.0-4883.gd61b0886.el9 2024-09-12T06:26:57.807 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4883.gd61b0886.el9 2024-09-12T06:26:57.807 INFO:teuthology.task.install:The correct ceph version 19.3.0-4883.gd61b0886 is installed. 2024-09-12T06:26:57.808 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d61b08865d4d98738f45ff2e4806d7fb6d03eca1 2024-09-12T06:26:57.952 DEBUG:teuthology.orchestra.run.smithi152:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-12T06:26:57.978 INFO:teuthology.orchestra.run.smithi152.stdout:19.3.0-4883.gd61b0886.el9 2024-09-12T06:26:57.978 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4883.gd61b0886.el9 2024-09-12T06:26:57.978 INFO:teuthology.task.install:The correct ceph version 19.3.0-4883.gd61b0886 is installed. 2024-09-12T06:26:57.980 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-12T06:26:57.980 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-12T06:26:57.980 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-12T06:26:58.005 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-12T06:26:58.005 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-12T06:26:58.047 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-12T06:26:58.048 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-12T06:26:58.048 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/usr/bin/daemon-helper 2024-09-12T06:26:58.072 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-12T06:26:58.137 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-12T06:26:58.138 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/usr/bin/daemon-helper 2024-09-12T06:26:58.167 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-12T06:26:58.235 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-12T06:26:58.236 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-12T06:26:58.236 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-12T06:26:58.260 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-12T06:26:58.325 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-12T06:26:58.325 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-12T06:26:58.355 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-12T06:26:58.422 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-12T06:26:58.423 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-12T06:26:58.423 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/usr/bin/stdin-killer 2024-09-12T06:26:58.448 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-12T06:26:58.513 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-12T06:26:58.513 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/usr/bin/stdin-killer 2024-09-12T06:26:58.538 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-12T06:26:58.604 INFO:teuthology.run_tasks:Running task ceph... 2024-09-12T06:26:58.690 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-12T06:26:58.690 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod 777 /var/log/ceph 2024-09-12T06:26:58.693 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod 777 /var/log/ceph 2024-09-12T06:26:58.717 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-12T06:26:58.718 DEBUG:teuthology.orchestra.run.smithi096:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-12T06:26:58.757 DEBUG:teuthology.orchestra.run.smithi152:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-12T06:26:58.783 INFO:tasks.ceph:Creating extra log directories... 2024-09-12T06:26:58.783 DEBUG:teuthology.orchestra.run.smithi096:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-12T06:26:58.822 DEBUG:teuthology.orchestra.run.smithi152:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-12T06:26:58.863 INFO:tasks.ceph:Creating ceph cluster cluster1... 2024-09-12T06:26:58.863 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-12T06:26:58.863 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/dis-2024-09-12_04:46:10-rbd-wip-dis-testing-distro-default-smithi/7902364', 'branch': 'wip-dis-testing', 'description': 'rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{centos_latest} workloads/rbd-mirror-stress-workunit-min-compat-client-octopus}', 'email': 'idryomov@gmail.com', 'first_in_suite': False, 'job_id': '7902364', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'dis-2024-09-12_04:46:10-rbd-wip-dis-testing-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-dis-testing'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'd61b08865d4d98738f45ff2e4806d7fb6d03eca1'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'd61b08865d4d98738f45ff2e4806d7fb6d03eca1'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-dis-testing', 'sha1': 'd61b08865d4d98738f45ff2e4806d7fb6d03eca1'}}, 'owner': 'scheduled_dis@teuthology', 'priority': 70, 'repo': 'https://github.com/ceph/ceph-ci', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 1680, 'sha1': 'd61b08865d4d98738f45ff2e4806d7fb6d03eca1', 'sleep_before_teardown': 0, 'subset': '42/100', 'suite': 'rbd', 'suite_branch': 'wip-dis-testing', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_d61b08865d4d98738f45ff2e4806d7fb6d03eca1/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci', 'suite_sha1': 'd61b08865d4d98738f45ff2e4806d7fb6d03eca1', 'targets': {'smithi096.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF5PABoXj0XD5QU3fRJj4OmIZGRIOMCS4DbBo8Dsdz7RZ6jxRAn4d86LXkqXIH+TJbhbJ5M+gn089Cqb92T+yFI=', 'smithi152.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBH6VISxqa8TYG4EzTv7+6lZ5JEAW0zEQsltSRY1l4Q2cK9o7oz4wS80C5VOQs74JCgsa8UBd2xTcMSPKgaUvUvI='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': 'd61b08865d4d98738f45ff2e4806d7fb6d03eca1'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'd61b08865d4d98738f45ff2e4806d7fb6d03eca1'}}, {'ceph': {'cluster': 'cluster2'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'RBD_MIRROR_MIN_COMPAT_CLIENT': 'octopus', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1'}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-12_04:46:10', 'tube': 'smithi', 'user': 'dis', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-12T06:26:58.864 DEBUG:teuthology.orchestra.run.smithi096:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2024-09-12T06:26:58.887 DEBUG:teuthology.orchestra.run.smithi152:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2024-09-12T06:26:58.907 DEBUG:teuthology.orchestra.run.smithi096:> sudo install -d -m0777 -- /var/run/ceph 2024-09-12T06:26:58.942 DEBUG:teuthology.orchestra.run.smithi152:> sudo install -d -m0777 -- /var/run/ceph 2024-09-12T06:26:58.972 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-12T06:26:58.972 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/scratch_devs of=/dev/stdout 2024-09-12T06:26:59.019 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-12T06:26:59.019 DEBUG:teuthology.orchestra.run.smithi096:> stat /dev/vg_nvme/lv_1 2024-09-12T06:26:59.075 INFO:teuthology.orchestra.run.smithi096.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-12T06:26:59.075 INFO:teuthology.orchestra.run.smithi096.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-12T06:26:59.075 INFO:teuthology.orchestra.run.smithi096.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-09-12T06:26:59.075 INFO:teuthology.orchestra.run.smithi096.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-12T06:26:59.075 INFO:teuthology.orchestra.run.smithi096.stdout:Context: system_u:object_r:device_t:s0 2024-09-12T06:26:59.075 INFO:teuthology.orchestra.run.smithi096.stdout:Access: 2024-09-12 06:26:53.376077818 +0000 2024-09-12T06:26:59.075 INFO:teuthology.orchestra.run.smithi096.stdout:Modify: 2024-09-12 06:23:48.094528526 +0000 2024-09-12T06:26:59.075 INFO:teuthology.orchestra.run.smithi096.stdout:Change: 2024-09-12 06:23:48.094528526 +0000 2024-09-12T06:26:59.075 INFO:teuthology.orchestra.run.smithi096.stdout: Birth: 2024-09-12 06:23:48.094528526 +0000 2024-09-12T06:26:59.076 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-12T06:26:59.137 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records in 2024-09-12T06:26:59.138 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records out 2024-09-12T06:26:59.138 INFO:teuthology.orchestra.run.smithi096.stderr:512 bytes copied, 0.000184644 s, 2.8 MB/s 2024-09-12T06:26:59.139 DEBUG:teuthology.orchestra.run.smithi096:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-12T06:26:59.195 DEBUG:teuthology.orchestra.run.smithi096:> stat /dev/vg_nvme/lv_2 2024-09-12T06:26:59.250 INFO:teuthology.orchestra.run.smithi096.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-12T06:26:59.250 INFO:teuthology.orchestra.run.smithi096.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-12T06:26:59.250 INFO:teuthology.orchestra.run.smithi096.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-09-12T06:26:59.250 INFO:teuthology.orchestra.run.smithi096.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-12T06:26:59.250 INFO:teuthology.orchestra.run.smithi096.stdout:Context: system_u:object_r:device_t:s0 2024-09-12T06:26:59.250 INFO:teuthology.orchestra.run.smithi096.stdout:Access: 2024-09-12 06:26:53.376077818 +0000 2024-09-12T06:26:59.250 INFO:teuthology.orchestra.run.smithi096.stdout:Modify: 2024-09-12 06:23:48.421521243 +0000 2024-09-12T06:26:59.250 INFO:teuthology.orchestra.run.smithi096.stdout:Change: 2024-09-12 06:23:48.421521243 +0000 2024-09-12T06:26:59.250 INFO:teuthology.orchestra.run.smithi096.stdout: Birth: 2024-09-12 06:23:48.421521243 +0000 2024-09-12T06:26:59.250 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-12T06:26:59.312 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records in 2024-09-12T06:26:59.313 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records out 2024-09-12T06:26:59.313 INFO:teuthology.orchestra.run.smithi096.stderr:512 bytes copied, 0.000207069 s, 2.5 MB/s 2024-09-12T06:26:59.314 DEBUG:teuthology.orchestra.run.smithi096:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-12T06:26:59.369 DEBUG:teuthology.orchestra.run.smithi096:> stat /dev/vg_nvme/lv_3 2024-09-12T06:26:59.425 INFO:teuthology.orchestra.run.smithi096.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-12T06:26:59.426 INFO:teuthology.orchestra.run.smithi096.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-12T06:26:59.426 INFO:teuthology.orchestra.run.smithi096.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-09-12T06:26:59.426 INFO:teuthology.orchestra.run.smithi096.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-12T06:26:59.426 INFO:teuthology.orchestra.run.smithi096.stdout:Context: system_u:object_r:device_t:s0 2024-09-12T06:26:59.426 INFO:teuthology.orchestra.run.smithi096.stdout:Access: 2024-09-12 06:26:53.376077818 +0000 2024-09-12T06:26:59.426 INFO:teuthology.orchestra.run.smithi096.stdout:Modify: 2024-09-12 06:23:48.749513937 +0000 2024-09-12T06:26:59.426 INFO:teuthology.orchestra.run.smithi096.stdout:Change: 2024-09-12 06:23:48.749513937 +0000 2024-09-12T06:26:59.427 INFO:teuthology.orchestra.run.smithi096.stdout: Birth: 2024-09-12 06:23:48.749513937 +0000 2024-09-12T06:26:59.427 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-12T06:26:59.489 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records in 2024-09-12T06:26:59.489 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records out 2024-09-12T06:26:59.489 INFO:teuthology.orchestra.run.smithi096.stderr:512 bytes copied, 0.000211616 s, 2.4 MB/s 2024-09-12T06:26:59.490 DEBUG:teuthology.orchestra.run.smithi096:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-12T06:26:59.548 DEBUG:teuthology.orchestra.run.smithi096:> stat /dev/vg_nvme/lv_4 2024-09-12T06:26:59.602 INFO:teuthology.orchestra.run.smithi096.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-12T06:26:59.603 INFO:teuthology.orchestra.run.smithi096.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-12T06:26:59.603 INFO:teuthology.orchestra.run.smithi096.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-09-12T06:26:59.603 INFO:teuthology.orchestra.run.smithi096.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-12T06:26:59.603 INFO:teuthology.orchestra.run.smithi096.stdout:Context: system_u:object_r:device_t:s0 2024-09-12T06:26:59.603 INFO:teuthology.orchestra.run.smithi096.stdout:Access: 2024-09-12 06:26:53.377077796 +0000 2024-09-12T06:26:59.603 INFO:teuthology.orchestra.run.smithi096.stdout:Modify: 2024-09-12 06:23:49.076506654 +0000 2024-09-12T06:26:59.603 INFO:teuthology.orchestra.run.smithi096.stdout:Change: 2024-09-12 06:23:49.076506654 +0000 2024-09-12T06:26:59.603 INFO:teuthology.orchestra.run.smithi096.stdout: Birth: 2024-09-12 06:23:49.076506654 +0000 2024-09-12T06:26:59.603 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-12T06:26:59.666 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records in 2024-09-12T06:26:59.666 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records out 2024-09-12T06:26:59.666 INFO:teuthology.orchestra.run.smithi096.stderr:512 bytes copied, 0.000191079 s, 2.7 MB/s 2024-09-12T06:26:59.667 DEBUG:teuthology.orchestra.run.smithi096:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-12T06:26:59.722 INFO:tasks.ceph:osd dev map: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-12T06:26:59.723 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi096.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-12T06:26:59.723 INFO:tasks.ceph:Generating config... 2024-09-12T06:26:59.724 INFO:tasks.ceph:[client] debug rbd = 20 2024-09-12T06:26:59.724 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2024-09-12T06:26:59.724 INFO:tasks.ceph:[client] log to stderr = False 2024-09-12T06:26:59.724 INFO:tasks.ceph:[client] rbd default features = 125 2024-09-12T06:26:59.724 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-12T06:26:59.724 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-12T06:26:59.724 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-12T06:26:59.725 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-12T06:26:59.725 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-12T06:26:59.725 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-12T06:26:59.725 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-12T06:26:59.725 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-12T06:26:59.725 INFO:tasks.ceph:[client.mirror.4] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-12T06:26:59.725 INFO:tasks.ceph:[client.mirror.4] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-12T06:26:59.725 INFO:tasks.ceph:[client.mirror.5] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-12T06:26:59.725 INFO:tasks.ceph:[client.mirror.5] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-12T06:26:59.725 INFO:tasks.ceph:[client.mirror.6] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-12T06:26:59.725 INFO:tasks.ceph:[client.mirror.6] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-12T06:26:59.725 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-12T06:26:59.726 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-09-12T06:26:59.726 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-09-12T06:26:59.726 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-12T06:26:59.726 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-12T06:26:59.726 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-12T06:26:59.726 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-12T06:26:59.726 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-12T06:26:59.726 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-12T06:26:59.726 INFO:tasks.ceph:[osd] bluestore compression algorithm = lz4 2024-09-12T06:26:59.726 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-09-12T06:26:59.726 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-12T06:26:59.726 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-09-12T06:26:59.726 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-09-12T06:26:59.726 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-12T06:26:59.727 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-12T06:26:59.727 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-09-12T06:26:59.727 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-12T06:26:59.727 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-12T06:26:59.727 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-12T06:26:59.727 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-12T06:26:59.727 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-12T06:26:59.727 INFO:tasks.ceph:Setting up cluster1.mon.a... 2024-09-12T06:26:59.727 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster1.keyring 2024-09-12T06:26:59.821 INFO:teuthology.orchestra.run.smithi096.stdout:creating /etc/ceph/cluster1.keyring 2024-09-12T06:26:59.824 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster1.keyring 2024-09-12T06:26:59.865 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-09-12T06:26:59.930 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster1.mon.a', '172.21.15.96')] 2024-09-12T06:26:59.931 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.96', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2024-09-12T06:26:59.932 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-12T06:26:59.932 DEBUG:teuthology.orchestra.run.smithi096:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-12T06:26:59.986 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.96 --print /home/ubuntu/cephtest/cluster1.monmap 2024-09-12T06:27:00.065 INFO:teuthology.orchestra.run.smithi096.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster1.monmap 2024-09-12T06:27:00.066 INFO:teuthology.orchestra.run.smithi096.stdout:monmaptool: generated fsid eb40689d-c0e4-40e2-9b05-3a3a0fa6fbce 2024-09-12T06:27:00.066 INFO:teuthology.orchestra.run.smithi096.stdout:setting min_mon_release = quincy 2024-09-12T06:27:00.066 INFO:teuthology.orchestra.run.smithi096.stdout:epoch 0 2024-09-12T06:27:00.066 INFO:teuthology.orchestra.run.smithi096.stdout:fsid eb40689d-c0e4-40e2-9b05-3a3a0fa6fbce 2024-09-12T06:27:00.066 INFO:teuthology.orchestra.run.smithi096.stdout:last_changed 2024-09-12T06:27:00.068898+0000 2024-09-12T06:27:00.066 INFO:teuthology.orchestra.run.smithi096.stdout:created 2024-09-12T06:27:00.068898+0000 2024-09-12T06:27:00.066 INFO:teuthology.orchestra.run.smithi096.stdout:min_mon_release 17 (quincy) 2024-09-12T06:27:00.066 INFO:teuthology.orchestra.run.smithi096.stdout:election_strategy: 1 2024-09-12T06:27:00.066 INFO:teuthology.orchestra.run.smithi096.stdout:0: [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] mon.a 2024-09-12T06:27:00.066 INFO:teuthology.orchestra.run.smithi096.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster1.monmap (1 monitors) 2024-09-12T06:27:00.067 DEBUG:teuthology.orchestra.run.smithi096:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-12T06:27:00.123 INFO:tasks.ceph:Writing /etc/ceph/cluster1.conf for FSID eb40689d-c0e4-40e2-9b05-3a3a0fa6fbce... 2024-09-12T06:27:00.124 DEBUG:teuthology.orchestra.run.smithi096:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2024-09-12T06:27:00.165 DEBUG:teuthology.orchestra.run.smithi152:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2024-09-12T06:27:00.206 INFO:teuthology.orchestra.run.smithi096.stdout:[global] 2024-09-12T06:27:00.206 INFO:teuthology.orchestra.run.smithi096.stdout: chdir = "" 2024-09-12T06:27:00.206 INFO:teuthology.orchestra.run.smithi096.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-12T06:27:00.206 INFO:teuthology.orchestra.run.smithi096.stdout: auth supported = cephx 2024-09-12T06:27:00.206 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:27:00.206 INFO:teuthology.orchestra.run.smithi096.stdout: filestore xattr use omap = true 2024-09-12T06:27:00.206 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:27:00.206 INFO:teuthology.orchestra.run.smithi096.stdout: mon clock drift allowed = 1.000 2024-09-12T06:27:00.206 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:27:00.206 INFO:teuthology.orchestra.run.smithi096.stdout: osd crush chooseleaf type = 0 2024-09-12T06:27:00.206 INFO:teuthology.orchestra.run.smithi096.stdout: auth debug = true 2024-09-12T06:27:00.207 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:27:00.207 INFO:teuthology.orchestra.run.smithi096.stdout: ms die on old message = true 2024-09-12T06:27:00.207 INFO:teuthology.orchestra.run.smithi096.stdout: ms die on bug = true 2024-09-12T06:27:00.207 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:27:00.207 INFO:teuthology.orchestra.run.smithi096.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-12T06:27:00.207 INFO:teuthology.orchestra.run.smithi096.stdout: mon pg warn max object skew = 0 2024-09-12T06:27:00.207 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:27:00.207 INFO:teuthology.orchestra.run.smithi096.stdout: # disable pg_autoscaler by default for new pools 2024-09-12T06:27:00.207 INFO:teuthology.orchestra.run.smithi096.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-12T06:27:00.207 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:27:00.207 INFO:teuthology.orchestra.run.smithi096.stdout: osd pool default size = 2 2024-09-12T06:27:00.207 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:27:00.207 INFO:teuthology.orchestra.run.smithi096.stdout: mon osd allow primary affinity = true 2024-09-12T06:27:00.207 INFO:teuthology.orchestra.run.smithi096.stdout: mon osd allow pg remap = true 2024-09-12T06:27:00.207 INFO:teuthology.orchestra.run.smithi096.stdout: mon warn on legacy crush tunables = false 2024-09-12T06:27:00.208 INFO:teuthology.orchestra.run.smithi096.stdout: mon warn on crush straw calc version zero = false 2024-09-12T06:27:00.208 INFO:teuthology.orchestra.run.smithi096.stdout: mon warn on no sortbitwise = false 2024-09-12T06:27:00.208 INFO:teuthology.orchestra.run.smithi096.stdout: mon warn on osd down out interval zero = false 2024-09-12T06:27:00.208 INFO:teuthology.orchestra.run.smithi096.stdout: mon warn on too few osds = false 2024-09-12T06:27:00.208 INFO:teuthology.orchestra.run.smithi096.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-12T06:27:00.208 INFO:teuthology.orchestra.run.smithi096.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-12T06:27:00.208 INFO:teuthology.orchestra.run.smithi096.stdout: mon_allow_pool_size_one = true 2024-09-12T06:27:00.208 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:27:00.208 INFO:teuthology.orchestra.run.smithi096.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-12T06:27:00.208 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:27:00.208 INFO:teuthology.orchestra.run.smithi096.stdout: osd default data pool replay window = 5 2024-09-12T06:27:00.208 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:27:00.208 INFO:teuthology.orchestra.run.smithi096.stdout: mon allow pool delete = true 2024-09-12T06:27:00.208 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:27:00.208 INFO:teuthology.orchestra.run.smithi096.stdout: mon cluster log file level = debug 2024-09-12T06:27:00.209 INFO:teuthology.orchestra.run.smithi096.stdout: debug asserts on shutdown = true 2024-09-12T06:27:00.209 INFO:teuthology.orchestra.run.smithi096.stdout: mon health detail to clog = false 2024-09-12T06:27:00.209 INFO:teuthology.orchestra.run.smithi096.stdout: mon host = 172.21.15.96 2024-09-12T06:27:00.209 INFO:teuthology.orchestra.run.smithi096.stdout: mon client directed command retry = 5 2024-09-12T06:27:00.209 INFO:teuthology.orchestra.run.smithi096.stdout: mon warn on pool no app = False 2024-09-12T06:27:00.209 INFO:teuthology.orchestra.run.smithi096.stdout: ms inject socket failures = 5000 2024-09-12T06:27:00.209 INFO:teuthology.orchestra.run.smithi096.stdout: fsid = eb40689d-c0e4-40e2-9b05-3a3a0fa6fbce 2024-09-12T06:27:00.209 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:27:00.209 INFO:teuthology.orchestra.run.smithi096.stdout:[osd] 2024-09-12T06:27:00.209 INFO:teuthology.orchestra.run.smithi096.stdout: osd journal size = 100 2024-09-12T06:27:00.209 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:27:00.209 INFO:teuthology.orchestra.run.smithi096.stdout: osd scrub load threshold = 5.0 2024-09-12T06:27:00.210 INFO:teuthology.orchestra.run.smithi096.stdout: osd scrub max interval = 600 2024-09-12T06:27:00.210 INFO:teuthology.orchestra.run.smithi096.stdout: osd mclock profile = high_recovery_ops 2024-09-12T06:27:00.210 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:27:00.210 INFO:teuthology.orchestra.run.smithi096.stdout: osd recover clone overlap = true 2024-09-12T06:27:00.210 INFO:teuthology.orchestra.run.smithi096.stdout: osd recovery max chunk = 1048576 2024-09-12T06:27:00.210 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:27:00.210 INFO:teuthology.orchestra.run.smithi096.stdout: osd debug shutdown = true 2024-09-12T06:27:00.210 INFO:teuthology.orchestra.run.smithi096.stdout: osd debug op order = true 2024-09-12T06:27:00.210 INFO:teuthology.orchestra.run.smithi096.stdout: osd debug verify stray on activate = true 2024-09-12T06:27:00.210 INFO:teuthology.orchestra.run.smithi096.stdout: osd debug trim objects = true 2024-09-12T06:27:00.210 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:27:00.210 INFO:teuthology.orchestra.run.smithi096.stdout: osd open classes on start = true 2024-09-12T06:27:00.210 INFO:teuthology.orchestra.run.smithi096.stdout: osd debug pg log writeout = true 2024-09-12T06:27:00.210 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:27:00.210 INFO:teuthology.orchestra.run.smithi096.stdout: osd deep scrub update digest min age = 30 2024-09-12T06:27:00.211 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:27:00.211 INFO:teuthology.orchestra.run.smithi096.stdout: osd map max advance = 10 2024-09-12T06:27:00.211 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:27:00.211 INFO:teuthology.orchestra.run.smithi096.stdout: journal zero on create = true 2024-09-12T06:27:00.211 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:27:00.211 INFO:teuthology.orchestra.run.smithi096.stdout: filestore ondisk finisher threads = 3 2024-09-12T06:27:00.211 INFO:teuthology.orchestra.run.smithi096.stdout: filestore apply finisher threads = 3 2024-09-12T06:27:00.211 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:27:00.211 INFO:teuthology.orchestra.run.smithi096.stdout: bdev debug aio = true 2024-09-12T06:27:00.211 INFO:teuthology.orchestra.run.smithi096.stdout: osd debug misdirected ops = true 2024-09-12T06:27:00.211 INFO:teuthology.orchestra.run.smithi096.stdout: bluestore block size = 96636764160 2024-09-12T06:27:00.211 INFO:teuthology.orchestra.run.smithi096.stdout: bluestore compression algorithm = lz4 2024-09-12T06:27:00.211 INFO:teuthology.orchestra.run.smithi096.stdout: bluestore compression mode = aggressive 2024-09-12T06:27:00.211 INFO:teuthology.orchestra.run.smithi096.stdout: bluestore fsck on mount = True 2024-09-12T06:27:00.211 INFO:teuthology.orchestra.run.smithi096.stdout: debug bluefs = 1/20 2024-09-12T06:27:00.211 INFO:teuthology.orchestra.run.smithi096.stdout: debug bluestore = 1/20 2024-09-12T06:27:00.212 INFO:teuthology.orchestra.run.smithi096.stdout: debug ms = 1 2024-09-12T06:27:00.212 INFO:teuthology.orchestra.run.smithi096.stdout: debug osd = 20 2024-09-12T06:27:00.212 INFO:teuthology.orchestra.run.smithi096.stdout: debug rocksdb = 4/10 2024-09-12T06:27:00.212 INFO:teuthology.orchestra.run.smithi096.stdout: mon osd backfillfull_ratio = 0.85 2024-09-12T06:27:00.212 INFO:teuthology.orchestra.run.smithi096.stdout: mon osd full ratio = 0.9 2024-09-12T06:27:00.212 INFO:teuthology.orchestra.run.smithi096.stdout: mon osd nearfull ratio = 0.8 2024-09-12T06:27:00.212 INFO:teuthology.orchestra.run.smithi096.stdout: osd failsafe full ratio = 0.95 2024-09-12T06:27:00.212 INFO:teuthology.orchestra.run.smithi096.stdout: osd objectstore = bluestore 2024-09-12T06:27:00.212 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:27:00.212 INFO:teuthology.orchestra.run.smithi096.stdout:[mgr] 2024-09-12T06:27:00.212 INFO:teuthology.orchestra.run.smithi096.stdout: debug ms = 1 2024-09-12T06:27:00.212 INFO:teuthology.orchestra.run.smithi096.stdout: debug mgr = 20 2024-09-12T06:27:00.212 INFO:teuthology.orchestra.run.smithi096.stdout: debug mon = 20 2024-09-12T06:27:00.213 INFO:teuthology.orchestra.run.smithi096.stdout: debug auth = 20 2024-09-12T06:27:00.213 INFO:teuthology.orchestra.run.smithi096.stdout: mon reweight min pgs per osd = 4 2024-09-12T06:27:00.213 INFO:teuthology.orchestra.run.smithi096.stdout: mon reweight min bytes per osd = 10 2024-09-12T06:27:00.213 INFO:teuthology.orchestra.run.smithi096.stdout: mgr/telemetry/nag = false 2024-09-12T06:27:00.213 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:27:00.213 INFO:teuthology.orchestra.run.smithi096.stdout:[mon] 2024-09-12T06:27:00.213 INFO:teuthology.orchestra.run.smithi096.stdout: debug ms = 1 2024-09-12T06:27:00.213 INFO:teuthology.orchestra.run.smithi096.stdout: debug mon = 20 2024-09-12T06:27:00.213 INFO:teuthology.orchestra.run.smithi096.stdout: debug paxos = 20 2024-09-12T06:27:00.213 INFO:teuthology.orchestra.run.smithi096.stdout: debug auth = 20 2024-09-12T06:27:00.213 INFO:teuthology.orchestra.run.smithi096.stdout: mon data avail warn = 5 2024-09-12T06:27:00.213 INFO:teuthology.orchestra.run.smithi096.stdout: mon mgr mkfs grace = 240 2024-09-12T06:27:00.213 INFO:teuthology.orchestra.run.smithi096.stdout: mon reweight min pgs per osd = 4 2024-09-12T06:27:00.213 INFO:teuthology.orchestra.run.smithi096.stdout: mon osd reporter subtree level = osd 2024-09-12T06:27:00.214 INFO:teuthology.orchestra.run.smithi096.stdout: mon osd prime pg temp = true 2024-09-12T06:27:00.214 INFO:teuthology.orchestra.run.smithi096.stdout: mon reweight min bytes per osd = 10 2024-09-12T06:27:00.214 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:27:00.214 INFO:teuthology.orchestra.run.smithi096.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-12T06:27:00.214 INFO:teuthology.orchestra.run.smithi096.stdout: auth mon ticket ttl = 660 # 11m 2024-09-12T06:27:00.214 INFO:teuthology.orchestra.run.smithi096.stdout: auth service ticket ttl = 240 # 4m 2024-09-12T06:27:00.214 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:27:00.214 INFO:teuthology.orchestra.run.smithi096.stdout: # don't complain about insecure global_id in the test suite 2024-09-12T06:27:00.214 INFO:teuthology.orchestra.run.smithi096.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-12T06:27:00.215 INFO:teuthology.orchestra.run.smithi096.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-12T06:27:00.215 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:27:00.215 INFO:teuthology.orchestra.run.smithi096.stdout: # 1m isn't quite enough 2024-09-12T06:27:00.215 INFO:teuthology.orchestra.run.smithi096.stdout: mon_down_mkfs_grace = 2m 2024-09-12T06:27:00.215 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:27:00.215 INFO:teuthology.orchestra.run.smithi096.stdout: mon_warn_on_filestore_osds = false 2024-09-12T06:27:00.215 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:27:00.215 INFO:teuthology.orchestra.run.smithi096.stdout:[client] 2024-09-12T06:27:00.215 INFO:teuthology.orchestra.run.smithi096.stdout: rgw cache enabled = true 2024-09-12T06:27:00.215 INFO:teuthology.orchestra.run.smithi096.stdout: rgw enable ops log = true 2024-09-12T06:27:00.215 INFO:teuthology.orchestra.run.smithi096.stdout: rgw enable usage log = true 2024-09-12T06:27:00.215 INFO:teuthology.orchestra.run.smithi096.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-12T06:27:00.215 INFO:teuthology.orchestra.run.smithi096.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-12T06:27:00.215 INFO:teuthology.orchestra.run.smithi096.stdout: debug rbd = 20 2024-09-12T06:27:00.216 INFO:teuthology.orchestra.run.smithi096.stdout: debug rbd_mirror = 15 2024-09-12T06:27:00.216 INFO:teuthology.orchestra.run.smithi096.stdout: log to stderr = False 2024-09-12T06:27:00.216 INFO:teuthology.orchestra.run.smithi096.stdout: rbd default features = 125 2024-09-12T06:27:00.216 INFO:teuthology.orchestra.run.smithi096.stdout:[mon.a] 2024-09-12T06:27:00.216 INFO:teuthology.orchestra.run.smithi096.stdout:[client.mirror.0] 2024-09-12T06:27:00.216 INFO:teuthology.orchestra.run.smithi096.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-12T06:27:00.216 INFO:teuthology.orchestra.run.smithi096.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-12T06:27:00.216 INFO:teuthology.orchestra.run.smithi096.stdout:[client.mirror.1] 2024-09-12T06:27:00.216 INFO:teuthology.orchestra.run.smithi096.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-12T06:27:00.216 INFO:teuthology.orchestra.run.smithi096.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-12T06:27:00.216 INFO:teuthology.orchestra.run.smithi096.stdout:[client.mirror.2] 2024-09-12T06:27:00.216 INFO:teuthology.orchestra.run.smithi096.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-12T06:27:00.216 INFO:teuthology.orchestra.run.smithi096.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-12T06:27:00.217 INFO:teuthology.orchestra.run.smithi096.stdout:[client.mirror.3] 2024-09-12T06:27:00.217 INFO:teuthology.orchestra.run.smithi096.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-12T06:27:00.217 INFO:teuthology.orchestra.run.smithi096.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-12T06:27:00.217 INFO:teuthology.orchestra.run.smithi096.stdout:[client.mirror.4] 2024-09-12T06:27:00.217 INFO:teuthology.orchestra.run.smithi096.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-12T06:27:00.217 INFO:teuthology.orchestra.run.smithi096.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-12T06:27:00.217 INFO:teuthology.orchestra.run.smithi096.stdout:[client.mirror.5] 2024-09-12T06:27:00.217 INFO:teuthology.orchestra.run.smithi096.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-12T06:27:00.217 INFO:teuthology.orchestra.run.smithi096.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-12T06:27:00.217 INFO:teuthology.orchestra.run.smithi096.stdout:[client.mirror.6] 2024-09-12T06:27:00.217 INFO:teuthology.orchestra.run.smithi096.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-12T06:27:00.217 INFO:teuthology.orchestra.run.smithi096.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-12T06:27:00.218 INFO:teuthology.orchestra.run.smithi152.stdout:[global] 2024-09-12T06:27:00.218 INFO:teuthology.orchestra.run.smithi152.stdout: chdir = "" 2024-09-12T06:27:00.218 INFO:teuthology.orchestra.run.smithi152.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-12T06:27:00.218 INFO:teuthology.orchestra.run.smithi152.stdout: auth supported = cephx 2024-09-12T06:27:00.218 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:27:00.218 INFO:teuthology.orchestra.run.smithi152.stdout: filestore xattr use omap = true 2024-09-12T06:27:00.218 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:27:00.218 INFO:teuthology.orchestra.run.smithi152.stdout: mon clock drift allowed = 1.000 2024-09-12T06:27:00.218 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:27:00.219 INFO:teuthology.orchestra.run.smithi152.stdout: osd crush chooseleaf type = 0 2024-09-12T06:27:00.219 INFO:teuthology.orchestra.run.smithi152.stdout: auth debug = true 2024-09-12T06:27:00.219 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:27:00.219 INFO:teuthology.orchestra.run.smithi152.stdout: ms die on old message = true 2024-09-12T06:27:00.219 INFO:teuthology.orchestra.run.smithi152.stdout: ms die on bug = true 2024-09-12T06:27:00.219 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:27:00.219 INFO:teuthology.orchestra.run.smithi152.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-12T06:27:00.219 INFO:teuthology.orchestra.run.smithi152.stdout: mon pg warn max object skew = 0 2024-09-12T06:27:00.219 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:27:00.219 INFO:teuthology.orchestra.run.smithi152.stdout: # disable pg_autoscaler by default for new pools 2024-09-12T06:27:00.219 INFO:teuthology.orchestra.run.smithi152.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-12T06:27:00.219 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:27:00.219 INFO:teuthology.orchestra.run.smithi152.stdout: osd pool default size = 2 2024-09-12T06:27:00.219 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:27:00.219 INFO:teuthology.orchestra.run.smithi152.stdout: mon osd allow primary affinity = true 2024-09-12T06:27:00.220 INFO:teuthology.orchestra.run.smithi152.stdout: mon osd allow pg remap = true 2024-09-12T06:27:00.220 INFO:teuthology.orchestra.run.smithi152.stdout: mon warn on legacy crush tunables = false 2024-09-12T06:27:00.220 INFO:teuthology.orchestra.run.smithi152.stdout: mon warn on crush straw calc version zero = false 2024-09-12T06:27:00.220 INFO:teuthology.orchestra.run.smithi152.stdout: mon warn on no sortbitwise = false 2024-09-12T06:27:00.220 INFO:teuthology.orchestra.run.smithi152.stdout: mon warn on osd down out interval zero = false 2024-09-12T06:27:00.220 INFO:teuthology.orchestra.run.smithi152.stdout: mon warn on too few osds = false 2024-09-12T06:27:00.220 INFO:teuthology.orchestra.run.smithi152.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-12T06:27:00.220 INFO:teuthology.orchestra.run.smithi152.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-12T06:27:00.220 INFO:teuthology.orchestra.run.smithi152.stdout: mon_allow_pool_size_one = true 2024-09-12T06:27:00.220 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:27:00.220 INFO:teuthology.orchestra.run.smithi152.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-12T06:27:00.220 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:27:00.220 INFO:teuthology.orchestra.run.smithi152.stdout: osd default data pool replay window = 5 2024-09-12T06:27:00.220 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:27:00.221 INFO:teuthology.orchestra.run.smithi152.stdout: mon allow pool delete = true 2024-09-12T06:27:00.221 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:27:00.221 INFO:teuthology.orchestra.run.smithi152.stdout: mon cluster log file level = debug 2024-09-12T06:27:00.221 INFO:teuthology.orchestra.run.smithi152.stdout: debug asserts on shutdown = true 2024-09-12T06:27:00.221 INFO:teuthology.orchestra.run.smithi152.stdout: mon health detail to clog = false 2024-09-12T06:27:00.221 INFO:teuthology.orchestra.run.smithi152.stdout: mon host = 172.21.15.96 2024-09-12T06:27:00.221 INFO:teuthology.orchestra.run.smithi152.stdout: mon client directed command retry = 5 2024-09-12T06:27:00.221 INFO:teuthology.orchestra.run.smithi152.stdout: mon warn on pool no app = False 2024-09-12T06:27:00.221 INFO:teuthology.orchestra.run.smithi152.stdout: ms inject socket failures = 5000 2024-09-12T06:27:00.221 INFO:teuthology.orchestra.run.smithi152.stdout: fsid = eb40689d-c0e4-40e2-9b05-3a3a0fa6fbce 2024-09-12T06:27:00.221 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:27:00.221 INFO:teuthology.orchestra.run.smithi152.stdout:[osd] 2024-09-12T06:27:00.221 INFO:teuthology.orchestra.run.smithi152.stdout: osd journal size = 100 2024-09-12T06:27:00.221 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:27:00.222 INFO:teuthology.orchestra.run.smithi152.stdout: osd scrub load threshold = 5.0 2024-09-12T06:27:00.222 INFO:teuthology.orchestra.run.smithi152.stdout: osd scrub max interval = 600 2024-09-12T06:27:00.222 INFO:teuthology.orchestra.run.smithi152.stdout: osd mclock profile = high_recovery_ops 2024-09-12T06:27:00.222 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:27:00.222 INFO:teuthology.orchestra.run.smithi152.stdout: osd recover clone overlap = true 2024-09-12T06:27:00.222 INFO:teuthology.orchestra.run.smithi152.stdout: osd recovery max chunk = 1048576 2024-09-12T06:27:00.222 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:27:00.222 INFO:teuthology.orchestra.run.smithi152.stdout: osd debug shutdown = true 2024-09-12T06:27:00.222 INFO:teuthology.orchestra.run.smithi152.stdout: osd debug op order = true 2024-09-12T06:27:00.222 INFO:teuthology.orchestra.run.smithi152.stdout: osd debug verify stray on activate = true 2024-09-12T06:27:00.222 INFO:teuthology.orchestra.run.smithi152.stdout: osd debug trim objects = true 2024-09-12T06:27:00.222 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:27:00.222 INFO:teuthology.orchestra.run.smithi152.stdout: osd open classes on start = true 2024-09-12T06:27:00.222 INFO:teuthology.orchestra.run.smithi152.stdout: osd debug pg log writeout = true 2024-09-12T06:27:00.222 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:27:00.222 INFO:teuthology.orchestra.run.smithi152.stdout: osd deep scrub update digest min age = 30 2024-09-12T06:27:00.223 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:27:00.223 INFO:teuthology.orchestra.run.smithi152.stdout: osd map max advance = 10 2024-09-12T06:27:00.223 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:27:00.223 INFO:teuthology.orchestra.run.smithi152.stdout: journal zero on create = true 2024-09-12T06:27:00.223 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:27:00.223 INFO:teuthology.orchestra.run.smithi152.stdout: filestore ondisk finisher threads = 3 2024-09-12T06:27:00.223 INFO:teuthology.orchestra.run.smithi152.stdout: filestore apply finisher threads = 3 2024-09-12T06:27:00.223 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:27:00.223 INFO:teuthology.orchestra.run.smithi152.stdout: bdev debug aio = true 2024-09-12T06:27:00.223 INFO:teuthology.orchestra.run.smithi152.stdout: osd debug misdirected ops = true 2024-09-12T06:27:00.223 INFO:teuthology.orchestra.run.smithi152.stdout: bluestore block size = 96636764160 2024-09-12T06:27:00.223 INFO:teuthology.orchestra.run.smithi152.stdout: bluestore compression algorithm = lz4 2024-09-12T06:27:00.224 INFO:teuthology.orchestra.run.smithi152.stdout: bluestore compression mode = aggressive 2024-09-12T06:27:00.224 INFO:teuthology.orchestra.run.smithi152.stdout: bluestore fsck on mount = True 2024-09-12T06:27:00.224 INFO:teuthology.orchestra.run.smithi152.stdout: debug bluefs = 1/20 2024-09-12T06:27:00.224 INFO:teuthology.orchestra.run.smithi152.stdout: debug bluestore = 1/20 2024-09-12T06:27:00.224 INFO:teuthology.orchestra.run.smithi152.stdout: debug ms = 1 2024-09-12T06:27:00.224 INFO:teuthology.orchestra.run.smithi152.stdout: debug osd = 20 2024-09-12T06:27:00.224 INFO:teuthology.orchestra.run.smithi152.stdout: debug rocksdb = 4/10 2024-09-12T06:27:00.224 INFO:teuthology.orchestra.run.smithi152.stdout: mon osd backfillfull_ratio = 0.85 2024-09-12T06:27:00.224 INFO:teuthology.orchestra.run.smithi152.stdout: mon osd full ratio = 0.9 2024-09-12T06:27:00.224 INFO:teuthology.orchestra.run.smithi152.stdout: mon osd nearfull ratio = 0.8 2024-09-12T06:27:00.224 INFO:teuthology.orchestra.run.smithi152.stdout: osd failsafe full ratio = 0.95 2024-09-12T06:27:00.224 INFO:teuthology.orchestra.run.smithi152.stdout: osd objectstore = bluestore 2024-09-12T06:27:00.224 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:27:00.224 INFO:teuthology.orchestra.run.smithi152.stdout:[mgr] 2024-09-12T06:27:00.224 INFO:teuthology.orchestra.run.smithi152.stdout: debug ms = 1 2024-09-12T06:27:00.225 INFO:teuthology.orchestra.run.smithi152.stdout: debug mgr = 20 2024-09-12T06:27:00.225 INFO:teuthology.orchestra.run.smithi152.stdout: debug mon = 20 2024-09-12T06:27:00.225 INFO:teuthology.orchestra.run.smithi152.stdout: debug auth = 20 2024-09-12T06:27:00.225 INFO:teuthology.orchestra.run.smithi152.stdout: mon reweight min pgs per osd = 4 2024-09-12T06:27:00.225 INFO:teuthology.orchestra.run.smithi152.stdout: mon reweight min bytes per osd = 10 2024-09-12T06:27:00.225 INFO:teuthology.orchestra.run.smithi152.stdout: mgr/telemetry/nag = false 2024-09-12T06:27:00.225 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:27:00.225 INFO:teuthology.orchestra.run.smithi152.stdout:[mon] 2024-09-12T06:27:00.225 INFO:teuthology.orchestra.run.smithi152.stdout: debug ms = 1 2024-09-12T06:27:00.225 INFO:teuthology.orchestra.run.smithi152.stdout: debug mon = 20 2024-09-12T06:27:00.225 INFO:teuthology.orchestra.run.smithi152.stdout: debug paxos = 20 2024-09-12T06:27:00.225 INFO:teuthology.orchestra.run.smithi152.stdout: debug auth = 20 2024-09-12T06:27:00.225 INFO:teuthology.orchestra.run.smithi152.stdout: mon data avail warn = 5 2024-09-12T06:27:00.225 INFO:teuthology.orchestra.run.smithi152.stdout: mon mgr mkfs grace = 240 2024-09-12T06:27:00.225 INFO:teuthology.orchestra.run.smithi152.stdout: mon reweight min pgs per osd = 4 2024-09-12T06:27:00.225 INFO:teuthology.orchestra.run.smithi152.stdout: mon osd reporter subtree level = osd 2024-09-12T06:27:00.226 INFO:teuthology.orchestra.run.smithi152.stdout: mon osd prime pg temp = true 2024-09-12T06:27:00.226 INFO:teuthology.orchestra.run.smithi152.stdout: mon reweight min bytes per osd = 10 2024-09-12T06:27:00.226 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:27:00.226 INFO:teuthology.orchestra.run.smithi152.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-12T06:27:00.226 INFO:teuthology.orchestra.run.smithi152.stdout: auth mon ticket ttl = 660 # 11m 2024-09-12T06:27:00.226 INFO:teuthology.orchestra.run.smithi152.stdout: auth service ticket ttl = 240 # 4m 2024-09-12T06:27:00.226 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:27:00.226 INFO:teuthology.orchestra.run.smithi152.stdout: # don't complain about insecure global_id in the test suite 2024-09-12T06:27:00.226 INFO:teuthology.orchestra.run.smithi152.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-12T06:27:00.226 INFO:teuthology.orchestra.run.smithi152.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-12T06:27:00.226 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:27:00.226 INFO:teuthology.orchestra.run.smithi152.stdout: # 1m isn't quite enough 2024-09-12T06:27:00.226 INFO:teuthology.orchestra.run.smithi152.stdout: mon_down_mkfs_grace = 2m 2024-09-12T06:27:00.226 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:27:00.226 INFO:teuthology.orchestra.run.smithi152.stdout: mon_warn_on_filestore_osds = false 2024-09-12T06:27:00.227 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:27:00.227 INFO:teuthology.orchestra.run.smithi152.stdout:[client] 2024-09-12T06:27:00.227 INFO:teuthology.orchestra.run.smithi152.stdout: rgw cache enabled = true 2024-09-12T06:27:00.227 INFO:teuthology.orchestra.run.smithi152.stdout: rgw enable ops log = true 2024-09-12T06:27:00.227 INFO:teuthology.orchestra.run.smithi152.stdout: rgw enable usage log = true 2024-09-12T06:27:00.227 INFO:teuthology.orchestra.run.smithi152.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-12T06:27:00.227 INFO:teuthology.orchestra.run.smithi152.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-12T06:27:00.227 INFO:teuthology.orchestra.run.smithi152.stdout: debug rbd = 20 2024-09-12T06:27:00.227 INFO:teuthology.orchestra.run.smithi152.stdout: debug rbd_mirror = 15 2024-09-12T06:27:00.228 INFO:teuthology.orchestra.run.smithi152.stdout: log to stderr = False 2024-09-12T06:27:00.228 INFO:teuthology.orchestra.run.smithi152.stdout: rbd default features = 125 2024-09-12T06:27:00.228 INFO:teuthology.orchestra.run.smithi152.stdout:[mon.a] 2024-09-12T06:27:00.228 INFO:teuthology.orchestra.run.smithi152.stdout:[client.mirror.0] 2024-09-12T06:27:00.228 INFO:teuthology.orchestra.run.smithi152.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-12T06:27:00.228 INFO:teuthology.orchestra.run.smithi152.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-12T06:27:00.228 INFO:teuthology.orchestra.run.smithi152.stdout:[client.mirror.1] 2024-09-12T06:27:00.228 INFO:teuthology.orchestra.run.smithi152.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-12T06:27:00.228 INFO:teuthology.orchestra.run.smithi152.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-12T06:27:00.228 INFO:teuthology.orchestra.run.smithi152.stdout:[client.mirror.2] 2024-09-12T06:27:00.228 INFO:teuthology.orchestra.run.smithi152.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-12T06:27:00.228 INFO:teuthology.orchestra.run.smithi152.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-12T06:27:00.228 INFO:teuthology.orchestra.run.smithi152.stdout:[client.mirror.3] 2024-09-12T06:27:00.228 INFO:teuthology.orchestra.run.smithi152.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-12T06:27:00.228 INFO:teuthology.orchestra.run.smithi152.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-12T06:27:00.228 INFO:teuthology.orchestra.run.smithi152.stdout:[client.mirror.4] 2024-09-12T06:27:00.229 INFO:teuthology.orchestra.run.smithi152.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-12T06:27:00.229 INFO:teuthology.orchestra.run.smithi152.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-12T06:27:00.229 INFO:teuthology.orchestra.run.smithi152.stdout:[client.mirror.5] 2024-09-12T06:27:00.229 INFO:teuthology.orchestra.run.smithi152.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-12T06:27:00.229 INFO:teuthology.orchestra.run.smithi152.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-12T06:27:00.229 INFO:teuthology.orchestra.run.smithi152.stdout:[client.mirror.6] 2024-09-12T06:27:00.229 INFO:teuthology.orchestra.run.smithi152.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-12T06:27:00.229 INFO:teuthology.orchestra.run.smithi152.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-12T06:27:00.230 INFO:tasks.ceph:Creating admin key on cluster1.mon.a... 2024-09-12T06:27:00.231 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster1.keyring 2024-09-12T06:27:00.324 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-12T06:27:00.325 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-12T06:27:00.325 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/etc/ceph/cluster1.keyring of=/dev/stdout 2024-09-12T06:27:00.340 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-12T06:27:00.341 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/home/ubuntu/cephtest/cluster1.monmap of=/dev/stdout 2024-09-12T06:27:00.396 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi096.front.sepia.ceph.com 2024-09-12T06:27:00.396 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-12T06:27:00.396 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/etc/ceph/cluster1.keyring 2024-09-12T06:27:00.396 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-09-12T06:27:00.470 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-12T06:27:00.470 DEBUG:teuthology.orchestra.run.smithi096:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2024-09-12T06:27:00.525 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi152.front.sepia.ceph.com 2024-09-12T06:27:00.525 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-12T06:27:00.525 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/etc/ceph/cluster1.keyring 2024-09-12T06:27:00.525 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-09-12T06:27:00.563 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-12T06:27:00.563 DEBUG:teuthology.orchestra.run.smithi152:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2024-09-12T06:27:00.620 INFO:tasks.ceph:Setting up mon nodes... 2024-09-12T06:27:00.620 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-12T06:27:00.620 DEBUG:teuthology.orchestra.run.smithi096:> sudo mkdir -p /var/lib/ceph/mgr/cluster1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster1-x/keyring 2024-09-12T06:27:00.671 INFO:teuthology.orchestra.run.smithi096.stdout:creating /var/lib/ceph/mgr/cluster1-x/keyring 2024-09-12T06:27:00.673 INFO:tasks.ceph:Setting up mds nodes... 2024-09-12T06:27:00.673 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-12T06:27:00.674 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster1.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.0.keyring 2024-09-12T06:27:00.714 INFO:teuthology.orchestra.run.smithi096.stdout:creating /etc/ceph/cluster1.client.0.keyring 2024-09-12T06:27:00.726 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster1.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.keyring 2024-09-12T06:27:00.775 INFO:teuthology.orchestra.run.smithi152.stdout:creating /etc/ceph/cluster1.client.mirror.keyring 2024-09-12T06:27:00.788 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster1.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.0.keyring 2024-09-12T06:27:00.830 INFO:teuthology.orchestra.run.smithi152.stdout:creating /etc/ceph/cluster1.client.mirror.0.keyring 2024-09-12T06:27:00.842 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster1.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.1.keyring 2024-09-12T06:27:00.924 INFO:teuthology.orchestra.run.smithi152.stdout:creating /etc/ceph/cluster1.client.mirror.1.keyring 2024-09-12T06:27:00.937 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster1.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.2.keyring 2024-09-12T06:27:01.017 INFO:teuthology.orchestra.run.smithi152.stdout:creating /etc/ceph/cluster1.client.mirror.2.keyring 2024-09-12T06:27:01.030 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster1.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.3.keyring 2024-09-12T06:27:01.110 INFO:teuthology.orchestra.run.smithi152.stdout:creating /etc/ceph/cluster1.client.mirror.3.keyring 2024-09-12T06:27:01.123 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster1.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.4.keyring 2024-09-12T06:27:01.205 INFO:teuthology.orchestra.run.smithi152.stdout:creating /etc/ceph/cluster1.client.mirror.4.keyring 2024-09-12T06:27:01.217 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster1.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.5.keyring 2024-09-12T06:27:01.299 INFO:teuthology.orchestra.run.smithi152.stdout:creating /etc/ceph/cluster1.client.mirror.5.keyring 2024-09-12T06:27:01.312 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster1.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.6.keyring 2024-09-12T06:27:01.392 INFO:teuthology.orchestra.run.smithi152.stdout:creating /etc/ceph/cluster1.client.mirror.6.keyring 2024-09-12T06:27:01.405 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-12T06:27:01.405 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi096.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-12T06:27:01.405 DEBUG:teuthology.orchestra.run.smithi096:> sudo mkdir -p /var/lib/ceph/osd/cluster1-0 2024-09-12T06:27:01.432 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-12T06:27:01.433 INFO:tasks.ceph:role: cluster1.osd.0 2024-09-12T06:27:01.433 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi096.front.sepia.ceph.com 2024-09-12T06:27:01.433 DEBUG:teuthology.orchestra.run.smithi096:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-12T06:27:01.506 INFO:teuthology.orchestra.run.smithi096.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-12T06:27:01.507 INFO:teuthology.orchestra.run.smithi096.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-12T06:27:01.507 INFO:teuthology.orchestra.run.smithi096.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-12T06:27:01.507 INFO:teuthology.orchestra.run.smithi096.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-12T06:27:01.507 INFO:teuthology.orchestra.run.smithi096.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-12T06:27:01.507 INFO:teuthology.orchestra.run.smithi096.stdout: = sunit=0 swidth=0 blks 2024-09-12T06:27:01.507 INFO:teuthology.orchestra.run.smithi096.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-12T06:27:01.507 INFO:teuthology.orchestra.run.smithi096.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-12T06:27:01.507 INFO:teuthology.orchestra.run.smithi096.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-12T06:27:01.507 INFO:teuthology.orchestra.run.smithi096.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-12T06:27:02.011 INFO:teuthology.orchestra.run.smithi096.stdout:Discarding blocks...Done. 2024-09-12T06:27:02.013 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi096.front.sepia.ceph.com -o noatime 2024-09-12T06:27:02.013 DEBUG:teuthology.orchestra.run.smithi096:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster1-0 2024-09-12T06:27:02.166 DEBUG:teuthology.orchestra.run.smithi096:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-0 2024-09-12T06:27:02.200 DEBUG:teuthology.orchestra.run.smithi096:> sudo mkdir -p /var/lib/ceph/osd/cluster1-1 2024-09-12T06:27:02.267 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-12T06:27:02.268 INFO:tasks.ceph:role: cluster1.osd.1 2024-09-12T06:27:02.268 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi096.front.sepia.ceph.com 2024-09-12T06:27:02.268 DEBUG:teuthology.orchestra.run.smithi096:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-12T06:27:02.341 INFO:teuthology.orchestra.run.smithi096.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-12T06:27:02.342 INFO:teuthology.orchestra.run.smithi096.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-12T06:27:02.342 INFO:teuthology.orchestra.run.smithi096.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-12T06:27:02.342 INFO:teuthology.orchestra.run.smithi096.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-12T06:27:02.342 INFO:teuthology.orchestra.run.smithi096.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-12T06:27:02.342 INFO:teuthology.orchestra.run.smithi096.stdout: = sunit=0 swidth=0 blks 2024-09-12T06:27:02.342 INFO:teuthology.orchestra.run.smithi096.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-12T06:27:02.342 INFO:teuthology.orchestra.run.smithi096.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-12T06:27:02.342 INFO:teuthology.orchestra.run.smithi096.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-12T06:27:02.342 INFO:teuthology.orchestra.run.smithi096.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-12T06:27:02.797 INFO:teuthology.orchestra.run.smithi096.stdout:Discarding blocks...Done. 2024-09-12T06:27:02.799 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi096.front.sepia.ceph.com -o noatime 2024-09-12T06:27:02.799 DEBUG:teuthology.orchestra.run.smithi096:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster1-1 2024-09-12T06:27:02.894 DEBUG:teuthology.orchestra.run.smithi096:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-1 2024-09-12T06:27:02.924 DEBUG:teuthology.orchestra.run.smithi096:> sudo mkdir -p /var/lib/ceph/osd/cluster1-2 2024-09-12T06:27:02.991 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-12T06:27:02.991 INFO:tasks.ceph:role: cluster1.osd.2 2024-09-12T06:27:02.991 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi096.front.sepia.ceph.com 2024-09-12T06:27:02.992 DEBUG:teuthology.orchestra.run.smithi096:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-12T06:27:03.065 INFO:teuthology.orchestra.run.smithi096.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-12T06:27:03.066 INFO:teuthology.orchestra.run.smithi096.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-12T06:27:03.066 INFO:teuthology.orchestra.run.smithi096.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-12T06:27:03.066 INFO:teuthology.orchestra.run.smithi096.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-12T06:27:03.066 INFO:teuthology.orchestra.run.smithi096.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-12T06:27:03.066 INFO:teuthology.orchestra.run.smithi096.stdout: = sunit=0 swidth=0 blks 2024-09-12T06:27:03.066 INFO:teuthology.orchestra.run.smithi096.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-12T06:27:03.066 INFO:teuthology.orchestra.run.smithi096.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-12T06:27:03.066 INFO:teuthology.orchestra.run.smithi096.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-12T06:27:03.066 INFO:teuthology.orchestra.run.smithi096.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-12T06:27:03.545 INFO:teuthology.orchestra.run.smithi096.stdout:Discarding blocks...Done. 2024-09-12T06:27:03.548 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi096.front.sepia.ceph.com -o noatime 2024-09-12T06:27:03.548 DEBUG:teuthology.orchestra.run.smithi096:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster1-2 2024-09-12T06:27:03.698 DEBUG:teuthology.orchestra.run.smithi096:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-2 2024-09-12T06:27:03.731 DEBUG:teuthology.orchestra.run.smithi096:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-09-12T06:27:03.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-09-12T06:27:03.850+0000 7f67fb01e640 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-0/keyring: can't open /var/lib/ceph/osd/cluster1-0/keyring: (2) No such file or directory 2024-09-12T06:27:03.851 INFO:teuthology.orchestra.run.smithi096.stderr:2024-09-12T06:27:03.852+0000 7f67fb01e640 -1 created new key in keyring /var/lib/ceph/osd/cluster1-0/keyring 2024-09-12T06:27:03.851 INFO:teuthology.orchestra.run.smithi096.stderr:2024-09-12T06:27:03.852+0000 7f67fb01e640 -1 bdev(0x5559afbb5800 /var/lib/ceph/osd/cluster1-0/block) open stat got: (1) Operation not permitted 2024-09-12T06:27:03.851 INFO:teuthology.orchestra.run.smithi096.stderr:2024-09-12T06:27:03.853+0000 7f67fb01e640 -1 bluestore(/var/lib/ceph/osd/cluster1-0) _read_fsid unparsable uuid 2024-09-12T06:27:07.028 DEBUG:teuthology.orchestra.run.smithi096:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-0 2024-09-12T06:27:07.058 DEBUG:teuthology.orchestra.run.smithi096:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-09-12T06:27:07.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-09-12T06:27:07.149+0000 7fbf526e4640 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-1/keyring: can't open /var/lib/ceph/osd/cluster1-1/keyring: (2) No such file or directory 2024-09-12T06:27:07.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-09-12T06:27:07.150+0000 7fbf526e4640 -1 created new key in keyring /var/lib/ceph/osd/cluster1-1/keyring 2024-09-12T06:27:07.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-09-12T06:27:07.150+0000 7fbf526e4640 -1 bdev(0x562c3276d800 /var/lib/ceph/osd/cluster1-1/block) open stat got: (1) Operation not permitted 2024-09-12T06:27:07.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-09-12T06:27:07.150+0000 7fbf526e4640 -1 bluestore(/var/lib/ceph/osd/cluster1-1) _read_fsid unparsable uuid 2024-09-12T06:27:10.546 DEBUG:teuthology.orchestra.run.smithi096:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-1 2024-09-12T06:27:10.575 DEBUG:teuthology.orchestra.run.smithi096:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-09-12T06:27:10.666 INFO:teuthology.orchestra.run.smithi096.stderr:2024-09-12T06:27:10.667+0000 7f406afb2640 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-2/keyring: can't open /var/lib/ceph/osd/cluster1-2/keyring: (2) No such file or directory 2024-09-12T06:27:10.666 INFO:teuthology.orchestra.run.smithi096.stderr:2024-09-12T06:27:10.668+0000 7f406afb2640 -1 created new key in keyring /var/lib/ceph/osd/cluster1-2/keyring 2024-09-12T06:27:10.666 INFO:teuthology.orchestra.run.smithi096.stderr:2024-09-12T06:27:10.668+0000 7f406afb2640 -1 bdev(0x55fa8b49f800 /var/lib/ceph/osd/cluster1-2/block) open stat got: (1) Operation not permitted 2024-09-12T06:27:10.666 INFO:teuthology.orchestra.run.smithi096.stderr:2024-09-12T06:27:10.668+0000 7f406afb2640 -1 bluestore(/var/lib/ceph/osd/cluster1-2) _read_fsid unparsable uuid 2024-09-12T06:27:13.828 DEBUG:teuthology.orchestra.run.smithi096:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-2 2024-09-12T06:27:13.856 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-12T06:27:13.856 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-12T06:27:13.856 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/var/lib/ceph/mgr/cluster1-x/keyring of=/dev/stdout 2024-09-12T06:27:13.923 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-12T06:27:13.923 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/var/lib/ceph/osd/cluster1-0/keyring of=/dev/stdout 2024-09-12T06:27:13.990 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-12T06:27:13.991 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/var/lib/ceph/osd/cluster1-1/keyring of=/dev/stdout 2024-09-12T06:27:14.057 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-12T06:27:14.057 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/var/lib/ceph/osd/cluster1-2/keyring of=/dev/stdout 2024-09-12T06:27:14.122 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-12T06:27:14.122 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/etc/ceph/cluster1.client.0.keyring of=/dev/stdout 2024-09-12T06:27:14.179 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-12T06:27:14.179 DEBUG:teuthology.orchestra.run.smithi152:> dd if=/etc/ceph/cluster1.client.mirror.keyring of=/dev/stdout 2024-09-12T06:27:14.199 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-12T06:27:14.199 DEBUG:teuthology.orchestra.run.smithi152:> dd if=/etc/ceph/cluster1.client.mirror.0.keyring of=/dev/stdout 2024-09-12T06:27:14.258 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-12T06:27:14.259 DEBUG:teuthology.orchestra.run.smithi152:> dd if=/etc/ceph/cluster1.client.mirror.1.keyring of=/dev/stdout 2024-09-12T06:27:14.316 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-12T06:27:14.317 DEBUG:teuthology.orchestra.run.smithi152:> dd if=/etc/ceph/cluster1.client.mirror.2.keyring of=/dev/stdout 2024-09-12T06:27:14.375 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-12T06:27:14.375 DEBUG:teuthology.orchestra.run.smithi152:> dd if=/etc/ceph/cluster1.client.mirror.3.keyring of=/dev/stdout 2024-09-12T06:27:14.433 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-12T06:27:14.433 DEBUG:teuthology.orchestra.run.smithi152:> dd if=/etc/ceph/cluster1.client.mirror.4.keyring of=/dev/stdout 2024-09-12T06:27:14.490 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-12T06:27:14.490 DEBUG:teuthology.orchestra.run.smithi152:> dd if=/etc/ceph/cluster1.client.mirror.5.keyring of=/dev/stdout 2024-09-12T06:27:14.547 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-12T06:27:14.547 DEBUG:teuthology.orchestra.run.smithi152:> dd if=/etc/ceph/cluster1.client.mirror.6.keyring of=/dev/stdout 2024-09-12T06:27:14.603 INFO:tasks.ceph:Adding keys to all mons... 2024-09-12T06:27:14.603 DEBUG:teuthology.orchestra.run.smithi096:> sudo tee -a /etc/ceph/cluster1.keyring 2024-09-12T06:27:14.629 INFO:teuthology.orchestra.run.smithi096.stdout:[mgr.x] 2024-09-12T06:27:14.629 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQC0ieJmdPAuKBAAmrZ2eiWFM0uwNnlReaLULA== 2024-09-12T06:27:14.629 INFO:teuthology.orchestra.run.smithi096.stdout:[osd.0] 2024-09-12T06:27:14.629 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQC3ieJmRErGMhAAwX4cmHARSjUYi9j3KwDSjw== 2024-09-12T06:27:14.629 INFO:teuthology.orchestra.run.smithi096.stdout:[osd.1] 2024-09-12T06:27:14.629 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQC7ieJm5df9CBAA+IWeKt3cJc1RJArPMk0A7g== 2024-09-12T06:27:14.629 INFO:teuthology.orchestra.run.smithi096.stdout:[osd.2] 2024-09-12T06:27:14.629 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQC+ieJm9N7bJxAALJulziHfZY3uzmDLfk192w== 2024-09-12T06:27:14.629 INFO:teuthology.orchestra.run.smithi096.stdout:[client.0] 2024-09-12T06:27:14.629 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQC0ieJmcu/CKhAAjZPDNdrKzDYfzaAPN/HvAQ== 2024-09-12T06:27:14.629 INFO:teuthology.orchestra.run.smithi096.stdout:[client.mirror] 2024-09-12T06:27:14.630 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQC0ieJmPwdoLhAAPLpLzaTwMyMvCwncGi2oNg== 2024-09-12T06:27:14.630 INFO:teuthology.orchestra.run.smithi096.stdout:[client.mirror.0] 2024-09-12T06:27:14.630 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQC0ieJmay+nMRAAf3FVmXMEA6ZvJq4h/jVJXw== 2024-09-12T06:27:14.630 INFO:teuthology.orchestra.run.smithi096.stdout:[client.mirror.1] 2024-09-12T06:27:14.630 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQC0ieJmCzlHNxAABrlTDk3zlswVIAQSeWfESQ== 2024-09-12T06:27:14.630 INFO:teuthology.orchestra.run.smithi096.stdout:[client.mirror.2] 2024-09-12T06:27:14.630 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQC1ieJm/iU/ARAAMLsUXSa4EFda8SRnJ/Xjhw== 2024-09-12T06:27:14.630 INFO:teuthology.orchestra.run.smithi096.stdout:[client.mirror.3] 2024-09-12T06:27:14.630 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQC1ieJmM57MBhAAJSABvLGi2OTN1Mr9zjRXbw== 2024-09-12T06:27:14.630 INFO:teuthology.orchestra.run.smithi096.stdout:[client.mirror.4] 2024-09-12T06:27:14.630 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQC1ieJmBN1sDBAAhFeKI2lQpheZO0/jobxfBA== 2024-09-12T06:27:14.630 INFO:teuthology.orchestra.run.smithi096.stdout:[client.mirror.5] 2024-09-12T06:27:14.630 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQC1ieJmFfcFEhAAI7D8pFJ2mx9KrurLZR2yPA== 2024-09-12T06:27:14.631 INFO:teuthology.orchestra.run.smithi096.stdout:[client.mirror.6] 2024-09-12T06:27:14.631 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQC1ieJm74GYFxAA2tCGUwBh9iCrmU+QZ7k90g== 2024-09-12T06:27:14.631 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-12T06:27:14.725 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-12T06:27:14.777 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-12T06:27:14.827 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-12T06:27:14.878 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-12T06:27:14.931 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-12T06:27:14.985 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-12T06:27:15.038 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-12T06:27:15.092 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-12T06:27:15.146 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-12T06:27:15.200 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-12T06:27:15.254 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-12T06:27:15.307 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-12T06:27:15.362 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-12T06:27:15.362 DEBUG:teuthology.orchestra.run.smithi096:> sudo mkdir -p /var/lib/ceph/mon/cluster1-a 2024-09-12T06:27:15.387 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster1 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster1.monmap --keyring /etc/ceph/cluster1.keyring 2024-09-12T06:27:15.482 DEBUG:teuthology.orchestra.run.smithi096:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster1-a 2024-09-12T06:27:15.506 DEBUG:teuthology.orchestra.run.smithi096:> rm -- /home/ubuntu/cephtest/cluster1.monmap 2024-09-12T06:27:15.562 INFO:tasks.ceph:Starting mon daemons in cluster cluster1... 2024-09-12T06:27:15.562 INFO:tasks.ceph.cluster1.mon.a:Restarting daemon 2024-09-12T06:27:15.563 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster1 -i a 2024-09-12T06:27:15.605 INFO:tasks.ceph.cluster1.mon.a:Started 2024-09-12T06:27:15.605 INFO:tasks.ceph:Starting mgr daemons in cluster cluster1... 2024-09-12T06:27:15.606 INFO:tasks.ceph.cluster1.mgr.x:Restarting daemon 2024-09-12T06:27:15.606 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster1 -i x 2024-09-12T06:27:15.608 INFO:tasks.ceph.cluster1.mgr.x:Started 2024-09-12T06:27:15.608 DEBUG:tasks.ceph:set 0 configs 2024-09-12T06:27:15.608 INFO:tasks.ceph:Setting crush tunables to default 2024-09-12T06:27:15.609 DEBUG:teuthology.orchestra.run.smithi096:> sudo ceph --cluster cluster1 osd crush tunables default 2024-09-12T06:27:15.779 INFO:teuthology.orchestra.run.smithi096.stderr:adjusted tunables profile to default 2024-09-12T06:27:15.790 INFO:tasks.ceph:check_enable_crimson: False 2024-09-12T06:27:15.790 INFO:tasks.ceph:Starting osd daemons in cluster cluster1... 2024-09-12T06:27:15.790 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-12T06:27:15.790 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/var/lib/ceph/osd/cluster1-0/fsid of=/dev/stdout 2024-09-12T06:27:15.814 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-12T06:27:15.815 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/var/lib/ceph/osd/cluster1-1/fsid of=/dev/stdout 2024-09-12T06:27:15.879 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-12T06:27:15.880 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/var/lib/ceph/osd/cluster1-2/fsid of=/dev/stdout 2024-09-12T06:27:15.944 DEBUG:teuthology.orchestra.run.smithi096:> sudo ceph --cluster cluster1 osd new eddb0881-a76f-48e4-a872-c9ca4ae8e4ee 0 2024-09-12T06:27:16.163 INFO:teuthology.orchestra.run.smithi096.stdout:0 2024-09-12T06:27:16.174 DEBUG:teuthology.orchestra.run.smithi096:> sudo ceph --cluster cluster1 osd new e312ae88-6495-4e04-9025-1b8e58a838eb 1 2024-09-12T06:27:16.352 INFO:teuthology.orchestra.run.smithi096.stdout:1 2024-09-12T06:27:16.363 DEBUG:teuthology.orchestra.run.smithi096:> sudo ceph --cluster cluster1 osd new f2b94172-d7bd-4308-baf0-d55d1f0c2790 2 2024-09-12T06:27:16.538 INFO:teuthology.orchestra.run.smithi096.stdout:2 2024-09-12T06:27:16.549 INFO:tasks.ceph.cluster1.osd.0:Restarting daemon 2024-09-12T06:27:16.550 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 0 2024-09-12T06:27:16.552 INFO:tasks.ceph.cluster1.osd.0:Started 2024-09-12T06:27:16.552 INFO:tasks.ceph.cluster1.osd.1:Restarting daemon 2024-09-12T06:27:16.552 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 1 2024-09-12T06:27:16.554 INFO:tasks.ceph.cluster1.osd.1:Started 2024-09-12T06:27:16.554 INFO:tasks.ceph.cluster1.osd.2:Restarting daemon 2024-09-12T06:27:16.555 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 2 2024-09-12T06:27:16.557 INFO:tasks.ceph.cluster1.osd.2:Started 2024-09-12T06:27:16.557 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-09-12T06:27:16.748 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:27:16.748 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":5,"fsid":"eb40689d-c0e4-40e2-9b05-3a3a0fa6fbce","created":"2024-09-12T06:27:15.694619+0000","modified":"2024-09-12T06:27:16.540973+0000","last_up_change":"0.000000","last_in_change":"2024-09-12T06:27:16.540973+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"eddb0881-a76f-48e4-a872-c9ca4ae8e4ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e312ae88-6495-4e04-9025-1b8e58a838eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f2b94172-d7bd-4308-baf0-d55d1f0c2790","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-12T06:27:16.759 INFO:tasks.ceph.ceph_manager.cluster1:[] 2024-09-12T06:27:16.759 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-12T06:27:17.061 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-09-12T06:27:17.355 INFO:teuthology.misc.health.smithi096.stderr:2024-09-12T06:27:17.357+0000 7fc6aeffd640 0 --2- 172.21.15.96:0/1194770023 >> v2:172.21.15.96:3300/0 conn(0x7fc6b0068f40 0x7fc6b00a6800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-12T06:27:17.423 INFO:teuthology.misc.health.smithi096.stdout: 2024-09-12T06:27:17.423 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":5,"fsid":"eb40689d-c0e4-40e2-9b05-3a3a0fa6fbce","created":"2024-09-12T06:27:15.694619+0000","modified":"2024-09-12T06:27:16.540973+0000","last_up_change":"0.000000","last_in_change":"2024-09-12T06:27:16.540973+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"eddb0881-a76f-48e4-a872-c9ca4ae8e4ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e312ae88-6495-4e04-9025-1b8e58a838eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f2b94172-d7bd-4308-baf0-d55d1f0c2790","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-12T06:27:17.432 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-12T06:27:17.745 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:27:17.747+0000 7f80e1d2e640 -1 Falling back to public interface 2024-09-12T06:27:17.763 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:27:17.765+0000 7fa336818640 -1 Falling back to public interface 2024-09-12T06:27:17.769 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:27:17.771+0000 7fac231c5640 -1 Falling back to public interface 2024-09-12T06:27:21.099 INFO:tasks.ceph.cluster1.mgr.x.smithi096.stderr:2024-09-12T06:27:21.101+0000 7fa3835f3640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-09-12T06:27:21.625 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:27:21.627+0000 7fac231c5640 -1 osd.2 0 log_to_monitors true 2024-09-12T06:27:21.776 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:27:21.778+0000 7f80e1d2e640 -1 osd.1 0 log_to_monitors true 2024-09-12T06:27:22.099 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:27:22.101+0000 7fa336818640 -1 osd.0 0 log_to_monitors true 2024-09-12T06:27:23.736 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-09-12T06:27:23.990 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:27:23.992+0000 7f80ddcbd640 -1 osd.1 0 waiting for initial osdmap 2024-09-12T06:27:23.998 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:27:24.000+0000 7fac1f968640 -1 osd.2 0 waiting for initial osdmap 2024-09-12T06:27:24.007 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:27:24.009+0000 7f80d5adc640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-12T06:27:24.012 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:27:24.014+0000 7fac16f73640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-12T06:27:24.081 INFO:teuthology.misc.health.smithi096.stdout: 2024-09-12T06:27:24.081 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":7,"fsid":"eb40689d-c0e4-40e2-9b05-3a3a0fa6fbce","created":"2024-09-12T06:27:15.694619+0000","modified":"2024-09-12T06:27:23.098205+0000","last_up_change":"0.000000","last_in_change":"2024-09-12T06:27:16.540973+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"eddb0881-a76f-48e4-a872-c9ca4ae8e4ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e312ae88-6495-4e04-9025-1b8e58a838eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f2b94172-d7bd-4308-baf0-d55d1f0c2790","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-12T06:27:24.092 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-12T06:27:24.773 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:27:24.775+0000 7fa3321e1640 -1 osd.0 0 waiting for initial osdmap 2024-09-12T06:27:24.787 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:27:24.789+0000 7fa32a000640 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-12T06:27:30.396 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-09-12T06:27:30.683 INFO:teuthology.misc.health.smithi096.stdout: 2024-09-12T06:27:30.683 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":11,"fsid":"eb40689d-c0e4-40e2-9b05-3a3a0fa6fbce","created":"2024-09-12T06:27:15.694619+0000","modified":"2024-09-12T06:27:27.123705+0000","last_up_change":"2024-09-12T06:27:25.120999+0000","last_in_change":"2024-09-12T06:27:16.540973+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-12T06:27:24.257197+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"eddb0881-a76f-48e4-a872-c9ca4ae8e4ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":3907677134},{"type":"v1","addr":"172.21.15.96:6809","nonce":3907677134}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":3907677134},{"type":"v1","addr":"172.21.15.96:6811","nonce":3907677134}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":3907677134},{"type":"v1","addr":"172.21.15.96:6815","nonce":3907677134}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":3907677134},{"type":"v1","addr":"172.21.15.96:6813","nonce":3907677134}]},"public_addr":"172.21.15.96:6809/3907677134","cluster_addr":"172.21.15.96:6811/3907677134","heartbeat_back_addr":"172.21.15.96:6815/3907677134","heartbeat_front_addr":"172.21.15.96:6813/3907677134","state":["exists","up"]},{"osd":1,"uuid":"e312ae88-6495-4e04-9025-1b8e58a838eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":3125814246},{"type":"v1","addr":"172.21.15.96:6801","nonce":3125814246}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":3125814246},{"type":"v1","addr":"172.21.15.96:6803","nonce":3125814246}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":3125814246},{"type":"v1","addr":"172.21.15.96:6807","nonce":3125814246}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":3125814246},{"type":"v1","addr":"172.21.15.96:6805","nonce":3125814246}]},"public_addr":"172.21.15.96:6801/3125814246","cluster_addr":"172.21.15.96:6803/3125814246","heartbeat_back_addr":"172.21.15.96:6807/3125814246","heartbeat_front_addr":"172.21.15.96:6805/3125814246","state":["exists","up"]},{"osd":2,"uuid":"f2b94172-d7bd-4308-baf0-d55d1f0c2790","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":1528565651},{"type":"v1","addr":"172.21.15.96:6817","nonce":1528565651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":1528565651},{"type":"v1","addr":"172.21.15.96:6819","nonce":1528565651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1528565651},{"type":"v1","addr":"172.21.15.96:6823","nonce":1528565651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":1528565651},{"type":"v1","addr":"172.21.15.96:6821","nonce":1528565651}]},"public_addr":"172.21.15.96:6817/1528565651","cluster_addr":"172.21.15.96:6819/1528565651","heartbeat_back_addr":"172.21.15.96:6823/1528565651","heartbeat_front_addr":"172.21.15.96:6821/1528565651","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T06:27:23.096668+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T06:27:22.767733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T06:27:22.668557+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-12T06:27:30.694 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-09-12T06:27:30.694 INFO:tasks.ceph:Creating RBD pool 2024-09-12T06:27:30.695 DEBUG:teuthology.orchestra.run.smithi096:> sudo ceph --cluster cluster1 osd pool create rbd 8 2024-09-12T06:27:31.224 INFO:teuthology.orchestra.run.smithi096.stderr:pool 'rbd' created 2024-09-12T06:27:31.238 DEBUG:teuthology.orchestra.run.smithi096:> rbd --cluster cluster1 pool init rbd 2024-09-12T06:27:34.246 INFO:tasks.ceph:Starting mds daemons in cluster cluster1... 2024-09-12T06:27:34.247 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 config log 1 --format=json 2024-09-12T06:27:34.247 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-12T06:27:34.541 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:27:34.553 INFO:teuthology.orchestra.run.smithi096.stdout:[{"version":4,"timestamp":"2024-09-12T06:27:24.776365+0000","name":"","changes":[{"name":"osd.0/osd_mclock_max_capacity_iops_ssd","new_value":"7718.531423"}]}] 2024-09-12T06:27:34.553 INFO:tasks.ceph_manager:config epoch is 4 2024-09-12T06:27:34.553 INFO:tasks.ceph:Waiting until cluster1 daemons up and pgs clean... 2024-09-12T06:27:34.553 INFO:tasks.ceph.ceph_manager.cluster1:waiting for mgr available 2024-09-12T06:27:34.554 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 mgr dump --format=json 2024-09-12T06:27:34.882 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:27:34.893 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":5,"flags":0,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":1811227623},{"type":"v1","addr":"172.21.15.96:6825","nonce":1811227623}]},"active_addr":"172.21.15.96:6825/1811227623","active_change":"2024-09-12T06:27:20.038798+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.96:0","nonce":1922250747}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.96:0","nonce":278095112}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.96:0","nonce":282574576}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.96:0","nonce":1592956356}]}]} 2024-09-12T06:27:34.895 INFO:tasks.ceph.ceph_manager.cluster1:mgr available! 2024-09-12T06:27:34.895 INFO:tasks.ceph.ceph_manager.cluster1:waiting for all up 2024-09-12T06:27:34.896 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-09-12T06:27:35.178 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:27:35.178 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":15,"fsid":"eb40689d-c0e4-40e2-9b05-3a3a0fa6fbce","created":"2024-09-12T06:27:15.694619+0000","modified":"2024-09-12T06:27:34.234185+0000","last_up_change":"2024-09-12T06:27:25.120999+0000","last_in_change":"2024-09-12T06:27:16.540973+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-12T06:27:24.257197+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-12T06:27:30.982769+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"eddb0881-a76f-48e4-a872-c9ca4ae8e4ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":3907677134},{"type":"v1","addr":"172.21.15.96:6809","nonce":3907677134}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":3907677134},{"type":"v1","addr":"172.21.15.96:6811","nonce":3907677134}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":3907677134},{"type":"v1","addr":"172.21.15.96:6815","nonce":3907677134}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":3907677134},{"type":"v1","addr":"172.21.15.96:6813","nonce":3907677134}]},"public_addr":"172.21.15.96:6809/3907677134","cluster_addr":"172.21.15.96:6811/3907677134","heartbeat_back_addr":"172.21.15.96:6815/3907677134","heartbeat_front_addr":"172.21.15.96:6813/3907677134","state":["exists","up"]},{"osd":1,"uuid":"e312ae88-6495-4e04-9025-1b8e58a838eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":3125814246},{"type":"v1","addr":"172.21.15.96:6801","nonce":3125814246}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":3125814246},{"type":"v1","addr":"172.21.15.96:6803","nonce":3125814246}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":3125814246},{"type":"v1","addr":"172.21.15.96:6807","nonce":3125814246}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":3125814246},{"type":"v1","addr":"172.21.15.96:6805","nonce":3125814246}]},"public_addr":"172.21.15.96:6801/3125814246","cluster_addr":"172.21.15.96:6803/3125814246","heartbeat_back_addr":"172.21.15.96:6807/3125814246","heartbeat_front_addr":"172.21.15.96:6805/3125814246","state":["exists","up"]},{"osd":2,"uuid":"f2b94172-d7bd-4308-baf0-d55d1f0c2790","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":1528565651},{"type":"v1","addr":"172.21.15.96:6817","nonce":1528565651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":1528565651},{"type":"v1","addr":"172.21.15.96:6819","nonce":1528565651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1528565651},{"type":"v1","addr":"172.21.15.96:6823","nonce":1528565651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":1528565651},{"type":"v1","addr":"172.21.15.96:6821","nonce":1528565651}]},"public_addr":"172.21.15.96:6817/1528565651","cluster_addr":"172.21.15.96:6819/1528565651","heartbeat_back_addr":"172.21.15.96:6823/1528565651","heartbeat_front_addr":"172.21.15.96:6821/1528565651","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T06:27:23.096668+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T06:27:22.767733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T06:27:22.668557+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-12T06:27:35.190 INFO:tasks.ceph.ceph_manager.cluster1:all up! 2024-09-12T06:27:35.190 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-09-12T06:27:35.475 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:27:35.475 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":15,"fsid":"eb40689d-c0e4-40e2-9b05-3a3a0fa6fbce","created":"2024-09-12T06:27:15.694619+0000","modified":"2024-09-12T06:27:34.234185+0000","last_up_change":"2024-09-12T06:27:25.120999+0000","last_in_change":"2024-09-12T06:27:16.540973+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-12T06:27:24.257197+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-12T06:27:30.982769+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"eddb0881-a76f-48e4-a872-c9ca4ae8e4ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":3907677134},{"type":"v1","addr":"172.21.15.96:6809","nonce":3907677134}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":3907677134},{"type":"v1","addr":"172.21.15.96:6811","nonce":3907677134}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":3907677134},{"type":"v1","addr":"172.21.15.96:6815","nonce":3907677134}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":3907677134},{"type":"v1","addr":"172.21.15.96:6813","nonce":3907677134}]},"public_addr":"172.21.15.96:6809/3907677134","cluster_addr":"172.21.15.96:6811/3907677134","heartbeat_back_addr":"172.21.15.96:6815/3907677134","heartbeat_front_addr":"172.21.15.96:6813/3907677134","state":["exists","up"]},{"osd":1,"uuid":"e312ae88-6495-4e04-9025-1b8e58a838eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":3125814246},{"type":"v1","addr":"172.21.15.96:6801","nonce":3125814246}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":3125814246},{"type":"v1","addr":"172.21.15.96:6803","nonce":3125814246}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":3125814246},{"type":"v1","addr":"172.21.15.96:6807","nonce":3125814246}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":3125814246},{"type":"v1","addr":"172.21.15.96:6805","nonce":3125814246}]},"public_addr":"172.21.15.96:6801/3125814246","cluster_addr":"172.21.15.96:6803/3125814246","heartbeat_back_addr":"172.21.15.96:6807/3125814246","heartbeat_front_addr":"172.21.15.96:6805/3125814246","state":["exists","up"]},{"osd":2,"uuid":"f2b94172-d7bd-4308-baf0-d55d1f0c2790","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":1528565651},{"type":"v1","addr":"172.21.15.96:6817","nonce":1528565651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":1528565651},{"type":"v1","addr":"172.21.15.96:6819","nonce":1528565651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1528565651},{"type":"v1","addr":"172.21.15.96:6823","nonce":1528565651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":1528565651},{"type":"v1","addr":"172.21.15.96:6821","nonce":1528565651}]},"public_addr":"172.21.15.96:6817/1528565651","cluster_addr":"172.21.15.96:6819/1528565651","heartbeat_back_addr":"172.21.15.96:6823/1528565651","heartbeat_front_addr":"172.21.15.96:6821/1528565651","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T06:27:23.096668+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T06:27:22.767733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T06:27:22.668557+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-12T06:27:35.487 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 flush_pg_stats 2024-09-12T06:27:35.488 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 flush_pg_stats 2024-09-12T06:27:35.488 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 flush_pg_stats 2024-09-12T06:27:35.624 INFO:teuthology.orchestra.run.smithi096.stdout:34359738373 2024-09-12T06:27:35.624 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2024-09-12T06:27:35.636 INFO:teuthology.orchestra.run.smithi096.stdout:34359738373 2024-09-12T06:27:35.637 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2024-09-12T06:27:35.637 INFO:teuthology.orchestra.run.smithi096.stdout:38654705669 2024-09-12T06:27:35.638 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2024-09-12T06:27:35.924 INFO:teuthology.orchestra.run.smithi096.stdout:34359738371 2024-09-12T06:27:35.937 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738371 for osd.1 2024-09-12T06:27:35.994 INFO:teuthology.orchestra.run.smithi096.stdout:34359738371 2024-09-12T06:27:35.998 INFO:teuthology.orchestra.run.smithi096.stdout:38654705668 2024-09-12T06:27:36.008 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738371 for osd.2 2024-09-12T06:27:36.011 INFO:tasks.ceph.ceph_manager.cluster1:need seq 38654705669 got 38654705668 for osd.0 2024-09-12T06:27:36.938 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2024-09-12T06:27:37.009 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2024-09-12T06:27:37.011 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2024-09-12T06:27:37.250 INFO:teuthology.orchestra.run.smithi096.stdout:34359738373 2024-09-12T06:27:37.263 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738373 for osd.1 2024-09-12T06:27:37.263 DEBUG:teuthology.parallel:result is None 2024-09-12T06:27:37.309 INFO:teuthology.orchestra.run.smithi096.stdout:34359738373 2024-09-12T06:27:37.316 INFO:teuthology.orchestra.run.smithi096.stdout:38654705669 2024-09-12T06:27:37.322 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738373 for osd.2 2024-09-12T06:27:37.323 DEBUG:teuthology.parallel:result is None 2024-09-12T06:27:37.328 INFO:tasks.ceph.ceph_manager.cluster1:need seq 38654705669 got 38654705669 for osd.0 2024-09-12T06:27:37.328 DEBUG:teuthology.parallel:result is None 2024-09-12T06:27:37.329 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2024-09-12T06:27:37.329 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-12T06:27:37.648 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:27:37.649 INFO:teuthology.orchestra.run.smithi096.stderr:dumped all 2024-09-12T06:27:37.660 INFO:teuthology.orchestra.run.smithi096.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-09-12T06:27:36.045012+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":81592,"kb_used_data":504,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283033928,"statfs":{"total":289910292480,"available":289826742272,"internally_reserved":0,"allocated":516096,"data_stored":1296850,"data_compressed":11786,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.817334"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-12T06:27:34.239270+0000","last_change":"2024-09-12T06:27:34.239517+0000","last_active":"2024-09-12T06:27:34.239270+0000","last_peered":"2024-09-12T06:27:34.239270+0000","last_clean":"2024-09-12T06:27:34.239270+0000","last_became_active":"2024-09-12T06:27:32.239188+0000","last_became_peered":"2024-09-12T06:27:32.239188+0000","last_unstale":"2024-09-12T06:27:34.239270+0000","last_undegraded":"2024-09-12T06:27:34.239270+0000","last_fullsized":"2024-09-12T06:27:34.239270+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_clean_scrub_stamp":"2024-09-12T06:27:31.224809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T15:03:24.767390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011638639999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-12T06:27:34.238899+0000","last_change":"2024-09-12T06:27:34.239133+0000","last_active":"2024-09-12T06:27:34.238899+0000","last_peered":"2024-09-12T06:27:34.238899+0000","last_clean":"2024-09-12T06:27:34.238899+0000","last_became_active":"2024-09-12T06:27:32.239159+0000","last_became_peered":"2024-09-12T06:27:32.239159+0000","last_unstale":"2024-09-12T06:27:34.238899+0000","last_undegraded":"2024-09-12T06:27:34.238899+0000","last_fullsized":"2024-09-12T06:27:34.238899+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_clean_scrub_stamp":"2024-09-12T06:27:31.224809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:25:39.727415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079824799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-12T06:27:34.238457+0000","last_change":"2024-09-12T06:27:34.238743+0000","last_active":"2024-09-12T06:27:34.238457+0000","last_peered":"2024-09-12T06:27:34.238457+0000","last_clean":"2024-09-12T06:27:34.238457+0000","last_became_active":"2024-09-12T06:27:32.239133+0000","last_became_peered":"2024-09-12T06:27:32.239133+0000","last_unstale":"2024-09-12T06:27:34.238457+0000","last_undegraded":"2024-09-12T06:27:34.238457+0000","last_fullsized":"2024-09-12T06:27:34.238457+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_clean_scrub_stamp":"2024-09-12T06:27:31.224809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:11:59.861727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050645400000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-12T06:27:34.238523+0000","last_change":"2024-09-12T06:27:34.238755+0000","last_active":"2024-09-12T06:27:34.238523+0000","last_peered":"2024-09-12T06:27:34.238523+0000","last_clean":"2024-09-12T06:27:34.238523+0000","last_became_active":"2024-09-12T06:27:32.239390+0000","last_became_peered":"2024-09-12T06:27:32.239390+0000","last_unstale":"2024-09-12T06:27:34.238523+0000","last_undegraded":"2024-09-12T06:27:34.238523+0000","last_fullsized":"2024-09-12T06:27:34.238523+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_clean_scrub_stamp":"2024-09-12T06:27:31.224809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:18:29.532273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050204899999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"15'2","reported_seq":22,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-12T06:27:34.243631+0000","last_change":"2024-09-12T06:27:34.242205+0000","last_active":"2024-09-12T06:27:34.243631+0000","last_peered":"2024-09-12T06:27:34.243631+0000","last_clean":"2024-09-12T06:27:34.243631+0000","last_became_active":"2024-09-12T06:27:32.240428+0000","last_became_peered":"2024-09-12T06:27:32.240428+0000","last_unstale":"2024-09-12T06:27:34.243631+0000","last_undegraded":"2024-09-12T06:27:34.243631+0000","last_fullsized":"2024-09-12T06:27:34.243631+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_clean_scrub_stamp":"2024-09-12T06:27:31.224809+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:37:32.701549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095364099999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-12T06:27:34.508328+0000","last_change":"2024-09-12T06:27:34.508520+0000","last_active":"2024-09-12T06:27:34.508328+0000","last_peered":"2024-09-12T06:27:34.508328+0000","last_clean":"2024-09-12T06:27:34.508328+0000","last_became_active":"2024-09-12T06:27:32.239409+0000","last_became_peered":"2024-09-12T06:27:32.239409+0000","last_unstale":"2024-09-12T06:27:34.508328+0000","last_undegraded":"2024-09-12T06:27:34.508328+0000","last_fullsized":"2024-09-12T06:27:34.508328+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_clean_scrub_stamp":"2024-09-12T06:27:31.224809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T10:21:29.326121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045469599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-12T06:27:34.508289+0000","last_change":"2024-09-12T06:27:34.508464+0000","last_active":"2024-09-12T06:27:34.508289+0000","last_peered":"2024-09-12T06:27:34.508289+0000","last_clean":"2024-09-12T06:27:34.508289+0000","last_became_active":"2024-09-12T06:27:32.239718+0000","last_became_peered":"2024-09-12T06:27:32.239718+0000","last_unstale":"2024-09-12T06:27:34.508289+0000","last_undegraded":"2024-09-12T06:27:34.508289+0000","last_fullsized":"2024-09-12T06:27:34.508289+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_clean_scrub_stamp":"2024-09-12T06:27:31.224809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:21:30.104428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039987200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"13'1","reported_seq":21,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-12T06:27:34.238844+0000","last_change":"2024-09-12T06:27:34.239177+0000","last_active":"2024-09-12T06:27:34.238844+0000","last_peered":"2024-09-12T06:27:34.238844+0000","last_clean":"2024-09-12T06:27:34.238844+0000","last_became_active":"2024-09-12T06:27:32.239878+0000","last_became_peered":"2024-09-12T06:27:32.239878+0000","last_unstale":"2024-09-12T06:27:34.238844+0000","last_undegraded":"2024-09-12T06:27:34.238844+0000","last_fullsized":"2024-09-12T06:27:34.238844+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_clean_scrub_stamp":"2024-09-12T06:27:31.224809+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:11:21.471227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091624399999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"10'52","reported_seq":88,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-12T06:27:34.238091+0000","last_change":"2024-09-12T06:27:27.078058+0000","last_active":"2024-09-12T06:27:34.238091+0000","last_peered":"2024-09-12T06:27:34.238091+0000","last_clean":"2024-09-12T06:27:34.238091+0000","last_became_active":"2024-09-12T06:27:27.076932+0000","last_became_peered":"2024-09-12T06:27:27.076932+0000","last_unstale":"2024-09-12T06:27:34.238091+0000","last_undegraded":"2024-09-12T06:27:34.238091+0000","last_fullsized":"2024-09-12T06:27:34.238091+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:27:25.120999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:27:25.120999+0000","last_clean_scrub_stamp":"2024-09-12T06:27:25.120999+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:20:08.653460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11786,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":9,"seq":38654705669,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":192512,"data_stored":629079,"data_compressed":5893,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":131072,"data_stored":38692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":192512,"data_stored":629079,"data_compressed":5893,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5893,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5893,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T06:27:37.661 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-12T06:27:37.941 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:27:37.941 INFO:teuthology.orchestra.run.smithi096.stderr:dumped all 2024-09-12T06:27:37.952 INFO:teuthology.orchestra.run.smithi096.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-09-12T06:27:36.045012+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":81592,"kb_used_data":504,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283033928,"statfs":{"total":289910292480,"available":289826742272,"internally_reserved":0,"allocated":516096,"data_stored":1296850,"data_compressed":11786,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.817334"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-12T06:27:34.239270+0000","last_change":"2024-09-12T06:27:34.239517+0000","last_active":"2024-09-12T06:27:34.239270+0000","last_peered":"2024-09-12T06:27:34.239270+0000","last_clean":"2024-09-12T06:27:34.239270+0000","last_became_active":"2024-09-12T06:27:32.239188+0000","last_became_peered":"2024-09-12T06:27:32.239188+0000","last_unstale":"2024-09-12T06:27:34.239270+0000","last_undegraded":"2024-09-12T06:27:34.239270+0000","last_fullsized":"2024-09-12T06:27:34.239270+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_clean_scrub_stamp":"2024-09-12T06:27:31.224809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T15:03:24.767390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011638639999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-12T06:27:34.238899+0000","last_change":"2024-09-12T06:27:34.239133+0000","last_active":"2024-09-12T06:27:34.238899+0000","last_peered":"2024-09-12T06:27:34.238899+0000","last_clean":"2024-09-12T06:27:34.238899+0000","last_became_active":"2024-09-12T06:27:32.239159+0000","last_became_peered":"2024-09-12T06:27:32.239159+0000","last_unstale":"2024-09-12T06:27:34.238899+0000","last_undegraded":"2024-09-12T06:27:34.238899+0000","last_fullsized":"2024-09-12T06:27:34.238899+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_clean_scrub_stamp":"2024-09-12T06:27:31.224809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:25:39.727415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079824799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-12T06:27:34.238457+0000","last_change":"2024-09-12T06:27:34.238743+0000","last_active":"2024-09-12T06:27:34.238457+0000","last_peered":"2024-09-12T06:27:34.238457+0000","last_clean":"2024-09-12T06:27:34.238457+0000","last_became_active":"2024-09-12T06:27:32.239133+0000","last_became_peered":"2024-09-12T06:27:32.239133+0000","last_unstale":"2024-09-12T06:27:34.238457+0000","last_undegraded":"2024-09-12T06:27:34.238457+0000","last_fullsized":"2024-09-12T06:27:34.238457+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_clean_scrub_stamp":"2024-09-12T06:27:31.224809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:11:59.861727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050645400000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-12T06:27:34.238523+0000","last_change":"2024-09-12T06:27:34.238755+0000","last_active":"2024-09-12T06:27:34.238523+0000","last_peered":"2024-09-12T06:27:34.238523+0000","last_clean":"2024-09-12T06:27:34.238523+0000","last_became_active":"2024-09-12T06:27:32.239390+0000","last_became_peered":"2024-09-12T06:27:32.239390+0000","last_unstale":"2024-09-12T06:27:34.238523+0000","last_undegraded":"2024-09-12T06:27:34.238523+0000","last_fullsized":"2024-09-12T06:27:34.238523+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_clean_scrub_stamp":"2024-09-12T06:27:31.224809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:18:29.532273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050204899999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"15'2","reported_seq":22,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-12T06:27:34.243631+0000","last_change":"2024-09-12T06:27:34.242205+0000","last_active":"2024-09-12T06:27:34.243631+0000","last_peered":"2024-09-12T06:27:34.243631+0000","last_clean":"2024-09-12T06:27:34.243631+0000","last_became_active":"2024-09-12T06:27:32.240428+0000","last_became_peered":"2024-09-12T06:27:32.240428+0000","last_unstale":"2024-09-12T06:27:34.243631+0000","last_undegraded":"2024-09-12T06:27:34.243631+0000","last_fullsized":"2024-09-12T06:27:34.243631+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_clean_scrub_stamp":"2024-09-12T06:27:31.224809+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:37:32.701549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095364099999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-12T06:27:34.508328+0000","last_change":"2024-09-12T06:27:34.508520+0000","last_active":"2024-09-12T06:27:34.508328+0000","last_peered":"2024-09-12T06:27:34.508328+0000","last_clean":"2024-09-12T06:27:34.508328+0000","last_became_active":"2024-09-12T06:27:32.239409+0000","last_became_peered":"2024-09-12T06:27:32.239409+0000","last_unstale":"2024-09-12T06:27:34.508328+0000","last_undegraded":"2024-09-12T06:27:34.508328+0000","last_fullsized":"2024-09-12T06:27:34.508328+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_clean_scrub_stamp":"2024-09-12T06:27:31.224809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T10:21:29.326121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045469599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-12T06:27:34.508289+0000","last_change":"2024-09-12T06:27:34.508464+0000","last_active":"2024-09-12T06:27:34.508289+0000","last_peered":"2024-09-12T06:27:34.508289+0000","last_clean":"2024-09-12T06:27:34.508289+0000","last_became_active":"2024-09-12T06:27:32.239718+0000","last_became_peered":"2024-09-12T06:27:32.239718+0000","last_unstale":"2024-09-12T06:27:34.508289+0000","last_undegraded":"2024-09-12T06:27:34.508289+0000","last_fullsized":"2024-09-12T06:27:34.508289+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_clean_scrub_stamp":"2024-09-12T06:27:31.224809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:21:30.104428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039987200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"13'1","reported_seq":21,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-12T06:27:34.238844+0000","last_change":"2024-09-12T06:27:34.239177+0000","last_active":"2024-09-12T06:27:34.238844+0000","last_peered":"2024-09-12T06:27:34.238844+0000","last_clean":"2024-09-12T06:27:34.238844+0000","last_became_active":"2024-09-12T06:27:32.239878+0000","last_became_peered":"2024-09-12T06:27:32.239878+0000","last_unstale":"2024-09-12T06:27:34.238844+0000","last_undegraded":"2024-09-12T06:27:34.238844+0000","last_fullsized":"2024-09-12T06:27:34.238844+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_clean_scrub_stamp":"2024-09-12T06:27:31.224809+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:11:21.471227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091624399999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"10'52","reported_seq":88,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-12T06:27:34.238091+0000","last_change":"2024-09-12T06:27:27.078058+0000","last_active":"2024-09-12T06:27:34.238091+0000","last_peered":"2024-09-12T06:27:34.238091+0000","last_clean":"2024-09-12T06:27:34.238091+0000","last_became_active":"2024-09-12T06:27:27.076932+0000","last_became_peered":"2024-09-12T06:27:27.076932+0000","last_unstale":"2024-09-12T06:27:34.238091+0000","last_undegraded":"2024-09-12T06:27:34.238091+0000","last_fullsized":"2024-09-12T06:27:34.238091+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:27:25.120999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:27:25.120999+0000","last_clean_scrub_stamp":"2024-09-12T06:27:25.120999+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:20:08.653460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11786,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":9,"seq":38654705669,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":192512,"data_stored":629079,"data_compressed":5893,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":131072,"data_stored":38692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":192512,"data_stored":629079,"data_compressed":5893,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5893,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5893,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T06:27:37.953 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2024-09-12T06:27:37.953 INFO:tasks.ceph:Waiting until ceph cluster cluster1 is healthy... 2024-09-12T06:27:37.953 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy 2024-09-12T06:27:37.954 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 health --format=json 2024-09-12T06:27:38.277 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:27:38.277 INFO:teuthology.orchestra.run.smithi096.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-12T06:27:38.290 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy done 2024-09-12T06:27:38.290 INFO:teuthology.run_tasks:Running task ceph... 2024-09-12T06:27:38.299 INFO:tasks.ceph:Creating ceph cluster cluster2... 2024-09-12T06:27:38.299 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-12T06:27:38.299 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/dis-2024-09-12_04:46:10-rbd-wip-dis-testing-distro-default-smithi/7902364', 'branch': 'wip-dis-testing', 'description': 'rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{centos_latest} workloads/rbd-mirror-stress-workunit-min-compat-client-octopus}', 'email': 'idryomov@gmail.com', 'first_in_suite': False, 'job_id': '7902364', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'dis-2024-09-12_04:46:10-rbd-wip-dis-testing-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-dis-testing'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'd61b08865d4d98738f45ff2e4806d7fb6d03eca1'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'd61b08865d4d98738f45ff2e4806d7fb6d03eca1'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-dis-testing', 'sha1': 'd61b08865d4d98738f45ff2e4806d7fb6d03eca1'}}, 'owner': 'scheduled_dis@teuthology', 'priority': 70, 'repo': 'https://github.com/ceph/ceph-ci', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 1680, 'sha1': 'd61b08865d4d98738f45ff2e4806d7fb6d03eca1', 'sleep_before_teardown': 0, 'subset': '42/100', 'suite': 'rbd', 'suite_branch': 'wip-dis-testing', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_d61b08865d4d98738f45ff2e4806d7fb6d03eca1/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci', 'suite_sha1': 'd61b08865d4d98738f45ff2e4806d7fb6d03eca1', 'targets': {'smithi096.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF5PABoXj0XD5QU3fRJj4OmIZGRIOMCS4DbBo8Dsdz7RZ6jxRAn4d86LXkqXIH+TJbhbJ5M+gn089Cqb92T+yFI=', 'smithi152.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBH6VISxqa8TYG4EzTv7+6lZ5JEAW0zEQsltSRY1l4Q2cK9o7oz4wS80C5VOQs74JCgsa8UBd2xTcMSPKgaUvUvI='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': 'd61b08865d4d98738f45ff2e4806d7fb6d03eca1'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'd61b08865d4d98738f45ff2e4806d7fb6d03eca1'}}, {'ceph': {'cluster': 'cluster2', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'd61b08865d4d98738f45ff2e4806d7fb6d03eca1'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'RBD_MIRROR_MIN_COMPAT_CLIENT': 'octopus', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1'}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-12_04:46:10', 'tube': 'smithi', 'user': 'dis', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-12T06:27:38.300 DEBUG:teuthology.orchestra.run.smithi096:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2024-09-12T06:27:38.302 DEBUG:teuthology.orchestra.run.smithi152:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2024-09-12T06:27:38.321 DEBUG:teuthology.orchestra.run.smithi096:> sudo install -d -m0777 -- /var/run/ceph 2024-09-12T06:27:38.358 DEBUG:teuthology.orchestra.run.smithi152:> sudo install -d -m0777 -- /var/run/ceph 2024-09-12T06:27:38.390 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-12T06:27:38.391 DEBUG:teuthology.orchestra.run.smithi152:> dd if=/scratch_devs of=/dev/stdout 2024-09-12T06:27:38.449 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-12T06:27:38.449 DEBUG:teuthology.orchestra.run.smithi152:> stat /dev/vg_nvme/lv_1 2024-09-12T06:27:38.506 INFO:teuthology.orchestra.run.smithi152.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-12T06:27:38.506 INFO:teuthology.orchestra.run.smithi152.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-12T06:27:38.506 INFO:teuthology.orchestra.run.smithi152.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-09-12T06:27:38.506 INFO:teuthology.orchestra.run.smithi152.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-12T06:27:38.506 INFO:teuthology.orchestra.run.smithi152.stdout:Context: system_u:object_r:device_t:s0 2024-09-12T06:27:38.506 INFO:teuthology.orchestra.run.smithi152.stdout:Access: 2024-09-12 06:26:27.134323726 +0000 2024-09-12T06:27:38.506 INFO:teuthology.orchestra.run.smithi152.stdout:Modify: 2024-09-12 06:23:48.100737148 +0000 2024-09-12T06:27:38.506 INFO:teuthology.orchestra.run.smithi152.stdout:Change: 2024-09-12 06:23:48.100737148 +0000 2024-09-12T06:27:38.506 INFO:teuthology.orchestra.run.smithi152.stdout: Birth: 2024-09-12 06:23:48.100737148 +0000 2024-09-12T06:27:38.507 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-12T06:27:38.571 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records in 2024-09-12T06:27:38.572 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records out 2024-09-12T06:27:38.572 INFO:teuthology.orchestra.run.smithi152.stderr:512 bytes copied, 0.000199048 s, 2.6 MB/s 2024-09-12T06:27:38.573 DEBUG:teuthology.orchestra.run.smithi152:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-12T06:27:38.631 DEBUG:teuthology.orchestra.run.smithi152:> stat /dev/vg_nvme/lv_2 2024-09-12T06:27:38.687 INFO:teuthology.orchestra.run.smithi152.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-12T06:27:38.687 INFO:teuthology.orchestra.run.smithi152.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-12T06:27:38.687 INFO:teuthology.orchestra.run.smithi152.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-09-12T06:27:38.687 INFO:teuthology.orchestra.run.smithi152.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-12T06:27:38.688 INFO:teuthology.orchestra.run.smithi152.stdout:Context: system_u:object_r:device_t:s0 2024-09-12T06:27:38.688 INFO:teuthology.orchestra.run.smithi152.stdout:Access: 2024-09-12 06:26:27.134323726 +0000 2024-09-12T06:27:38.688 INFO:teuthology.orchestra.run.smithi152.stdout:Modify: 2024-09-12 06:23:48.419733548 +0000 2024-09-12T06:27:38.688 INFO:teuthology.orchestra.run.smithi152.stdout:Change: 2024-09-12 06:23:48.419733548 +0000 2024-09-12T06:27:38.688 INFO:teuthology.orchestra.run.smithi152.stdout: Birth: 2024-09-12 06:23:48.419733548 +0000 2024-09-12T06:27:38.688 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-12T06:27:38.752 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records in 2024-09-12T06:27:38.753 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records out 2024-09-12T06:27:38.753 INFO:teuthology.orchestra.run.smithi152.stderr:512 bytes copied, 0.00022733 s, 2.3 MB/s 2024-09-12T06:27:38.754 DEBUG:teuthology.orchestra.run.smithi152:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-12T06:27:38.811 DEBUG:teuthology.orchestra.run.smithi152:> stat /dev/vg_nvme/lv_3 2024-09-12T06:27:38.867 INFO:teuthology.orchestra.run.smithi152.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-12T06:27:38.867 INFO:teuthology.orchestra.run.smithi152.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-12T06:27:38.867 INFO:teuthology.orchestra.run.smithi152.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-09-12T06:27:38.867 INFO:teuthology.orchestra.run.smithi152.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-12T06:27:38.867 INFO:teuthology.orchestra.run.smithi152.stdout:Context: system_u:object_r:device_t:s0 2024-09-12T06:27:38.867 INFO:teuthology.orchestra.run.smithi152.stdout:Access: 2024-09-12 06:26:27.134323726 +0000 2024-09-12T06:27:38.867 INFO:teuthology.orchestra.run.smithi152.stdout:Modify: 2024-09-12 06:23:48.744729880 +0000 2024-09-12T06:27:38.867 INFO:teuthology.orchestra.run.smithi152.stdout:Change: 2024-09-12 06:23:48.744729880 +0000 2024-09-12T06:27:38.867 INFO:teuthology.orchestra.run.smithi152.stdout: Birth: 2024-09-12 06:23:48.744729880 +0000 2024-09-12T06:27:38.867 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-12T06:27:38.931 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records in 2024-09-12T06:27:38.932 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records out 2024-09-12T06:27:38.932 INFO:teuthology.orchestra.run.smithi152.stderr:512 bytes copied, 0.000301199 s, 1.7 MB/s 2024-09-12T06:27:38.933 DEBUG:teuthology.orchestra.run.smithi152:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-12T06:27:38.990 DEBUG:teuthology.orchestra.run.smithi152:> stat /dev/vg_nvme/lv_4 2024-09-12T06:27:39.046 INFO:teuthology.orchestra.run.smithi152.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-12T06:27:39.046 INFO:teuthology.orchestra.run.smithi152.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-12T06:27:39.046 INFO:teuthology.orchestra.run.smithi152.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-09-12T06:27:39.046 INFO:teuthology.orchestra.run.smithi152.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-12T06:27:39.046 INFO:teuthology.orchestra.run.smithi152.stdout:Context: system_u:object_r:device_t:s0 2024-09-12T06:27:39.047 INFO:teuthology.orchestra.run.smithi152.stdout:Access: 2024-09-12 06:26:27.135323715 +0000 2024-09-12T06:27:39.047 INFO:teuthology.orchestra.run.smithi152.stdout:Modify: 2024-09-12 06:23:49.066726246 +0000 2024-09-12T06:27:39.047 INFO:teuthology.orchestra.run.smithi152.stdout:Change: 2024-09-12 06:23:49.066726246 +0000 2024-09-12T06:27:39.047 INFO:teuthology.orchestra.run.smithi152.stdout: Birth: 2024-09-12 06:23:49.066726246 +0000 2024-09-12T06:27:39.047 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-12T06:27:39.110 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records in 2024-09-12T06:27:39.110 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records out 2024-09-12T06:27:39.111 INFO:teuthology.orchestra.run.smithi152.stderr:512 bytes copied, 0.000230237 s, 2.2 MB/s 2024-09-12T06:27:39.111 DEBUG:teuthology.orchestra.run.smithi152:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-12T06:27:39.168 INFO:tasks.ceph:osd dev map: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-12T06:27:39.168 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi152.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-12T06:27:39.168 INFO:tasks.ceph:Generating config... 2024-09-12T06:27:39.169 INFO:tasks.ceph:[client] debug rbd = 20 2024-09-12T06:27:39.169 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2024-09-12T06:27:39.169 INFO:tasks.ceph:[client] log to stderr = False 2024-09-12T06:27:39.169 INFO:tasks.ceph:[client] rbd default features = 125 2024-09-12T06:27:39.169 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-12T06:27:39.169 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-12T06:27:39.170 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-12T06:27:39.170 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-12T06:27:39.170 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-12T06:27:39.170 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-12T06:27:39.170 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-12T06:27:39.170 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-12T06:27:39.170 INFO:tasks.ceph:[client.mirror.4] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-12T06:27:39.170 INFO:tasks.ceph:[client.mirror.4] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-12T06:27:39.170 INFO:tasks.ceph:[client.mirror.5] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-12T06:27:39.170 INFO:tasks.ceph:[client.mirror.5] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-12T06:27:39.170 INFO:tasks.ceph:[client.mirror.6] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-12T06:27:39.171 INFO:tasks.ceph:[client.mirror.6] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-12T06:27:39.171 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-12T06:27:39.171 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-09-12T06:27:39.171 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-09-12T06:27:39.171 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-12T06:27:39.171 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-12T06:27:39.171 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-12T06:27:39.171 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-12T06:27:39.171 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-12T06:27:39.171 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-12T06:27:39.171 INFO:tasks.ceph:[osd] bluestore compression algorithm = lz4 2024-09-12T06:27:39.171 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-09-12T06:27:39.171 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-12T06:27:39.171 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-09-12T06:27:39.171 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-09-12T06:27:39.172 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-12T06:27:39.172 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-12T06:27:39.172 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-09-12T06:27:39.172 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-12T06:27:39.172 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-12T06:27:39.172 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-12T06:27:39.172 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-12T06:27:39.172 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-12T06:27:39.172 INFO:tasks.ceph:Setting up cluster2.mon.a... 2024-09-12T06:27:39.172 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster2.keyring 2024-09-12T06:27:39.247 INFO:teuthology.orchestra.run.smithi152.stdout:creating /etc/ceph/cluster2.keyring 2024-09-12T06:27:39.250 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster2.keyring 2024-09-12T06:27:39.332 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-09-12T06:27:39.397 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster2.mon.a', '172.21.15.152')] 2024-09-12T06:27:39.397 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.152', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2024-09-12T06:27:39.398 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-12T06:27:39.398 DEBUG:teuthology.orchestra.run.smithi152:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-12T06:27:39.452 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.152 --print /home/ubuntu/cephtest/cluster2.monmap 2024-09-12T06:27:39.531 INFO:teuthology.orchestra.run.smithi152.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster2.monmap 2024-09-12T06:27:39.532 INFO:teuthology.orchestra.run.smithi152.stdout:monmaptool: generated fsid cccfff5c-d879-49a4-9cf1-b06db11d0b7e 2024-09-12T06:27:39.532 INFO:teuthology.orchestra.run.smithi152.stdout:setting min_mon_release = quincy 2024-09-12T06:27:39.532 INFO:teuthology.orchestra.run.smithi152.stdout:epoch 0 2024-09-12T06:27:39.532 INFO:teuthology.orchestra.run.smithi152.stdout:fsid cccfff5c-d879-49a4-9cf1-b06db11d0b7e 2024-09-12T06:27:39.532 INFO:teuthology.orchestra.run.smithi152.stdout:last_changed 2024-09-12T06:27:39.534774+0000 2024-09-12T06:27:39.532 INFO:teuthology.orchestra.run.smithi152.stdout:created 2024-09-12T06:27:39.534774+0000 2024-09-12T06:27:39.532 INFO:teuthology.orchestra.run.smithi152.stdout:min_mon_release 17 (quincy) 2024-09-12T06:27:39.532 INFO:teuthology.orchestra.run.smithi152.stdout:election_strategy: 1 2024-09-12T06:27:39.532 INFO:teuthology.orchestra.run.smithi152.stdout:0: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.a 2024-09-12T06:27:39.532 INFO:teuthology.orchestra.run.smithi152.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster2.monmap (1 monitors) 2024-09-12T06:27:39.533 DEBUG:teuthology.orchestra.run.smithi152:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-12T06:27:39.589 INFO:tasks.ceph:Writing /etc/ceph/cluster2.conf for FSID cccfff5c-d879-49a4-9cf1-b06db11d0b7e... 2024-09-12T06:27:39.590 DEBUG:teuthology.orchestra.run.smithi096:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2024-09-12T06:27:39.592 DEBUG:teuthology.orchestra.run.smithi152:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2024-09-12T06:27:39.639 INFO:teuthology.orchestra.run.smithi096.stdout:[global] 2024-09-12T06:27:39.639 INFO:teuthology.orchestra.run.smithi096.stdout: chdir = "" 2024-09-12T06:27:39.639 INFO:teuthology.orchestra.run.smithi096.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-12T06:27:39.639 INFO:teuthology.orchestra.run.smithi096.stdout: auth supported = cephx 2024-09-12T06:27:39.640 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:27:39.640 INFO:teuthology.orchestra.run.smithi096.stdout: filestore xattr use omap = true 2024-09-12T06:27:39.640 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:27:39.640 INFO:teuthology.orchestra.run.smithi096.stdout: mon clock drift allowed = 1.000 2024-09-12T06:27:39.640 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:27:39.640 INFO:teuthology.orchestra.run.smithi096.stdout: osd crush chooseleaf type = 0 2024-09-12T06:27:39.640 INFO:teuthology.orchestra.run.smithi096.stdout: auth debug = true 2024-09-12T06:27:39.640 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:27:39.640 INFO:teuthology.orchestra.run.smithi096.stdout: ms die on old message = true 2024-09-12T06:27:39.640 INFO:teuthology.orchestra.run.smithi096.stdout: ms die on bug = true 2024-09-12T06:27:39.640 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:27:39.640 INFO:teuthology.orchestra.run.smithi096.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-12T06:27:39.640 INFO:teuthology.orchestra.run.smithi096.stdout: mon pg warn max object skew = 0 2024-09-12T06:27:39.640 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:27:39.640 INFO:teuthology.orchestra.run.smithi096.stdout: # disable pg_autoscaler by default for new pools 2024-09-12T06:27:39.641 INFO:teuthology.orchestra.run.smithi096.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-12T06:27:39.641 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:27:39.641 INFO:teuthology.orchestra.run.smithi096.stdout: osd pool default size = 2 2024-09-12T06:27:39.641 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:27:39.641 INFO:teuthology.orchestra.run.smithi096.stdout: mon osd allow primary affinity = true 2024-09-12T06:27:39.641 INFO:teuthology.orchestra.run.smithi096.stdout: mon osd allow pg remap = true 2024-09-12T06:27:39.641 INFO:teuthology.orchestra.run.smithi096.stdout: mon warn on legacy crush tunables = false 2024-09-12T06:27:39.641 INFO:teuthology.orchestra.run.smithi096.stdout: mon warn on crush straw calc version zero = false 2024-09-12T06:27:39.641 INFO:teuthology.orchestra.run.smithi096.stdout: mon warn on no sortbitwise = false 2024-09-12T06:27:39.641 INFO:teuthology.orchestra.run.smithi096.stdout: mon warn on osd down out interval zero = false 2024-09-12T06:27:39.641 INFO:teuthology.orchestra.run.smithi096.stdout: mon warn on too few osds = false 2024-09-12T06:27:39.641 INFO:teuthology.orchestra.run.smithi096.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-12T06:27:39.641 INFO:teuthology.orchestra.run.smithi096.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-12T06:27:39.641 INFO:teuthology.orchestra.run.smithi096.stdout: mon_allow_pool_size_one = true 2024-09-12T06:27:39.642 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:27:39.642 INFO:teuthology.orchestra.run.smithi096.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-12T06:27:39.642 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:27:39.642 INFO:teuthology.orchestra.run.smithi096.stdout: osd default data pool replay window = 5 2024-09-12T06:27:39.642 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:27:39.642 INFO:teuthology.orchestra.run.smithi096.stdout: mon allow pool delete = true 2024-09-12T06:27:39.642 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:27:39.642 INFO:teuthology.orchestra.run.smithi096.stdout: mon cluster log file level = debug 2024-09-12T06:27:39.642 INFO:teuthology.orchestra.run.smithi096.stdout: debug asserts on shutdown = true 2024-09-12T06:27:39.642 INFO:teuthology.orchestra.run.smithi096.stdout: mon health detail to clog = false 2024-09-12T06:27:39.642 INFO:teuthology.orchestra.run.smithi096.stdout: mon host = 172.21.15.152 2024-09-12T06:27:39.643 INFO:teuthology.orchestra.run.smithi096.stdout: mon client directed command retry = 5 2024-09-12T06:27:39.643 INFO:teuthology.orchestra.run.smithi096.stdout: mon warn on pool no app = False 2024-09-12T06:27:39.643 INFO:teuthology.orchestra.run.smithi096.stdout: ms inject socket failures = 5000 2024-09-12T06:27:39.643 INFO:teuthology.orchestra.run.smithi096.stdout: fsid = cccfff5c-d879-49a4-9cf1-b06db11d0b7e 2024-09-12T06:27:39.643 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:27:39.643 INFO:teuthology.orchestra.run.smithi096.stdout:[osd] 2024-09-12T06:27:39.643 INFO:teuthology.orchestra.run.smithi096.stdout: osd journal size = 100 2024-09-12T06:27:39.643 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:27:39.643 INFO:teuthology.orchestra.run.smithi096.stdout: osd scrub load threshold = 5.0 2024-09-12T06:27:39.643 INFO:teuthology.orchestra.run.smithi096.stdout: osd scrub max interval = 600 2024-09-12T06:27:39.643 INFO:teuthology.orchestra.run.smithi096.stdout: osd mclock profile = high_recovery_ops 2024-09-12T06:27:39.643 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:27:39.643 INFO:teuthology.orchestra.run.smithi096.stdout: osd recover clone overlap = true 2024-09-12T06:27:39.643 INFO:teuthology.orchestra.run.smithi096.stdout: osd recovery max chunk = 1048576 2024-09-12T06:27:39.643 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:27:39.643 INFO:teuthology.orchestra.run.smithi096.stdout: osd debug shutdown = true 2024-09-12T06:27:39.644 INFO:teuthology.orchestra.run.smithi096.stdout: osd debug op order = true 2024-09-12T06:27:39.644 INFO:teuthology.orchestra.run.smithi096.stdout: osd debug verify stray on activate = true 2024-09-12T06:27:39.644 INFO:teuthology.orchestra.run.smithi096.stdout: osd debug trim objects = true 2024-09-12T06:27:39.644 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:27:39.644 INFO:teuthology.orchestra.run.smithi096.stdout: osd open classes on start = true 2024-09-12T06:27:39.644 INFO:teuthology.orchestra.run.smithi096.stdout: osd debug pg log writeout = true 2024-09-12T06:27:39.644 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:27:39.644 INFO:teuthology.orchestra.run.smithi096.stdout: osd deep scrub update digest min age = 30 2024-09-12T06:27:39.644 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:27:39.644 INFO:teuthology.orchestra.run.smithi096.stdout: osd map max advance = 10 2024-09-12T06:27:39.644 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:27:39.644 INFO:teuthology.orchestra.run.smithi096.stdout: journal zero on create = true 2024-09-12T06:27:39.644 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:27:39.644 INFO:teuthology.orchestra.run.smithi096.stdout: filestore ondisk finisher threads = 3 2024-09-12T06:27:39.644 INFO:teuthology.orchestra.run.smithi096.stdout: filestore apply finisher threads = 3 2024-09-12T06:27:39.645 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:27:39.645 INFO:teuthology.orchestra.run.smithi096.stdout: bdev debug aio = true 2024-09-12T06:27:39.645 INFO:teuthology.orchestra.run.smithi096.stdout: osd debug misdirected ops = true 2024-09-12T06:27:39.645 INFO:teuthology.orchestra.run.smithi096.stdout: bluestore block size = 96636764160 2024-09-12T06:27:39.645 INFO:teuthology.orchestra.run.smithi096.stdout: bluestore compression algorithm = lz4 2024-09-12T06:27:39.645 INFO:teuthology.orchestra.run.smithi096.stdout: bluestore compression mode = aggressive 2024-09-12T06:27:39.645 INFO:teuthology.orchestra.run.smithi096.stdout: bluestore fsck on mount = True 2024-09-12T06:27:39.645 INFO:teuthology.orchestra.run.smithi096.stdout: debug bluefs = 1/20 2024-09-12T06:27:39.645 INFO:teuthology.orchestra.run.smithi096.stdout: debug bluestore = 1/20 2024-09-12T06:27:39.645 INFO:teuthology.orchestra.run.smithi096.stdout: debug ms = 1 2024-09-12T06:27:39.645 INFO:teuthology.orchestra.run.smithi096.stdout: debug osd = 20 2024-09-12T06:27:39.645 INFO:teuthology.orchestra.run.smithi096.stdout: debug rocksdb = 4/10 2024-09-12T06:27:39.645 INFO:teuthology.orchestra.run.smithi096.stdout: mon osd backfillfull_ratio = 0.85 2024-09-12T06:27:39.645 INFO:teuthology.orchestra.run.smithi096.stdout: mon osd full ratio = 0.9 2024-09-12T06:27:39.645 INFO:teuthology.orchestra.run.smithi096.stdout: mon osd nearfull ratio = 0.8 2024-09-12T06:27:39.646 INFO:teuthology.orchestra.run.smithi096.stdout: osd failsafe full ratio = 0.95 2024-09-12T06:27:39.646 INFO:teuthology.orchestra.run.smithi096.stdout: osd objectstore = bluestore 2024-09-12T06:27:39.646 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:27:39.646 INFO:teuthology.orchestra.run.smithi096.stdout:[mgr] 2024-09-12T06:27:39.646 INFO:teuthology.orchestra.run.smithi096.stdout: debug ms = 1 2024-09-12T06:27:39.646 INFO:teuthology.orchestra.run.smithi096.stdout: debug mgr = 20 2024-09-12T06:27:39.646 INFO:teuthology.orchestra.run.smithi096.stdout: debug mon = 20 2024-09-12T06:27:39.646 INFO:teuthology.orchestra.run.smithi096.stdout: debug auth = 20 2024-09-12T06:27:39.646 INFO:teuthology.orchestra.run.smithi096.stdout: mon reweight min pgs per osd = 4 2024-09-12T06:27:39.646 INFO:teuthology.orchestra.run.smithi096.stdout: mon reweight min bytes per osd = 10 2024-09-12T06:27:39.646 INFO:teuthology.orchestra.run.smithi096.stdout: mgr/telemetry/nag = false 2024-09-12T06:27:39.646 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:27:39.646 INFO:teuthology.orchestra.run.smithi096.stdout:[mon] 2024-09-12T06:27:39.647 INFO:teuthology.orchestra.run.smithi096.stdout: debug ms = 1 2024-09-12T06:27:39.647 INFO:teuthology.orchestra.run.smithi096.stdout: debug mon = 20 2024-09-12T06:27:39.647 INFO:teuthology.orchestra.run.smithi096.stdout: debug paxos = 20 2024-09-12T06:27:39.647 INFO:teuthology.orchestra.run.smithi096.stdout: debug auth = 20 2024-09-12T06:27:39.647 INFO:teuthology.orchestra.run.smithi096.stdout: mon data avail warn = 5 2024-09-12T06:27:39.647 INFO:teuthology.orchestra.run.smithi096.stdout: mon mgr mkfs grace = 240 2024-09-12T06:27:39.647 INFO:teuthology.orchestra.run.smithi096.stdout: mon reweight min pgs per osd = 4 2024-09-12T06:27:39.647 INFO:teuthology.orchestra.run.smithi096.stdout: mon osd reporter subtree level = osd 2024-09-12T06:27:39.647 INFO:teuthology.orchestra.run.smithi096.stdout: mon osd prime pg temp = true 2024-09-12T06:27:39.647 INFO:teuthology.orchestra.run.smithi096.stdout: mon reweight min bytes per osd = 10 2024-09-12T06:27:39.647 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:27:39.647 INFO:teuthology.orchestra.run.smithi096.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-12T06:27:39.647 INFO:teuthology.orchestra.run.smithi096.stdout: auth mon ticket ttl = 660 # 11m 2024-09-12T06:27:39.647 INFO:teuthology.orchestra.run.smithi096.stdout: auth service ticket ttl = 240 # 4m 2024-09-12T06:27:39.648 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:27:39.648 INFO:teuthology.orchestra.run.smithi096.stdout: # don't complain about insecure global_id in the test suite 2024-09-12T06:27:39.648 INFO:teuthology.orchestra.run.smithi096.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-12T06:27:39.648 INFO:teuthology.orchestra.run.smithi096.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-12T06:27:39.648 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:27:39.648 INFO:teuthology.orchestra.run.smithi096.stdout: # 1m isn't quite enough 2024-09-12T06:27:39.648 INFO:teuthology.orchestra.run.smithi096.stdout: mon_down_mkfs_grace = 2m 2024-09-12T06:27:39.648 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:27:39.648 INFO:teuthology.orchestra.run.smithi096.stdout: mon_warn_on_filestore_osds = false 2024-09-12T06:27:39.648 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:27:39.648 INFO:teuthology.orchestra.run.smithi096.stdout:[client] 2024-09-12T06:27:39.648 INFO:teuthology.orchestra.run.smithi096.stdout: rgw cache enabled = true 2024-09-12T06:27:39.648 INFO:teuthology.orchestra.run.smithi096.stdout: rgw enable ops log = true 2024-09-12T06:27:39.649 INFO:teuthology.orchestra.run.smithi096.stdout: rgw enable usage log = true 2024-09-12T06:27:39.649 INFO:teuthology.orchestra.run.smithi096.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-12T06:27:39.649 INFO:teuthology.orchestra.run.smithi096.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-12T06:27:39.649 INFO:teuthology.orchestra.run.smithi096.stdout: debug rbd = 20 2024-09-12T06:27:39.649 INFO:teuthology.orchestra.run.smithi096.stdout: debug rbd_mirror = 15 2024-09-12T06:27:39.649 INFO:teuthology.orchestra.run.smithi096.stdout: log to stderr = False 2024-09-12T06:27:39.649 INFO:teuthology.orchestra.run.smithi096.stdout: rbd default features = 125 2024-09-12T06:27:39.649 INFO:teuthology.orchestra.run.smithi096.stdout:[mon.a] 2024-09-12T06:27:39.649 INFO:teuthology.orchestra.run.smithi096.stdout:[client.mirror.0] 2024-09-12T06:27:39.649 INFO:teuthology.orchestra.run.smithi096.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-12T06:27:39.649 INFO:teuthology.orchestra.run.smithi096.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-12T06:27:39.649 INFO:teuthology.orchestra.run.smithi096.stdout:[client.mirror.1] 2024-09-12T06:27:39.649 INFO:teuthology.orchestra.run.smithi096.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-12T06:27:39.649 INFO:teuthology.orchestra.run.smithi096.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-12T06:27:39.650 INFO:teuthology.orchestra.run.smithi096.stdout:[client.mirror.2] 2024-09-12T06:27:39.650 INFO:teuthology.orchestra.run.smithi096.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-12T06:27:39.650 INFO:teuthology.orchestra.run.smithi096.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-12T06:27:39.650 INFO:teuthology.orchestra.run.smithi096.stdout:[client.mirror.3] 2024-09-12T06:27:39.650 INFO:teuthology.orchestra.run.smithi096.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-12T06:27:39.650 INFO:teuthology.orchestra.run.smithi096.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-12T06:27:39.650 INFO:teuthology.orchestra.run.smithi096.stdout:[client.mirror.4] 2024-09-12T06:27:39.650 INFO:teuthology.orchestra.run.smithi096.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-12T06:27:39.650 INFO:teuthology.orchestra.run.smithi096.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-12T06:27:39.650 INFO:teuthology.orchestra.run.smithi096.stdout:[client.mirror.5] 2024-09-12T06:27:39.650 INFO:teuthology.orchestra.run.smithi096.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-12T06:27:39.650 INFO:teuthology.orchestra.run.smithi096.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-12T06:27:39.650 INFO:teuthology.orchestra.run.smithi096.stdout:[client.mirror.6] 2024-09-12T06:27:39.650 INFO:teuthology.orchestra.run.smithi096.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-12T06:27:39.650 INFO:teuthology.orchestra.run.smithi096.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-12T06:27:39.671 INFO:teuthology.orchestra.run.smithi152.stdout:[global] 2024-09-12T06:27:39.671 INFO:teuthology.orchestra.run.smithi152.stdout: chdir = "" 2024-09-12T06:27:39.671 INFO:teuthology.orchestra.run.smithi152.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-12T06:27:39.672 INFO:teuthology.orchestra.run.smithi152.stdout: auth supported = cephx 2024-09-12T06:27:39.672 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:27:39.672 INFO:teuthology.orchestra.run.smithi152.stdout: filestore xattr use omap = true 2024-09-12T06:27:39.672 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:27:39.672 INFO:teuthology.orchestra.run.smithi152.stdout: mon clock drift allowed = 1.000 2024-09-12T06:27:39.672 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:27:39.672 INFO:teuthology.orchestra.run.smithi152.stdout: osd crush chooseleaf type = 0 2024-09-12T06:27:39.672 INFO:teuthology.orchestra.run.smithi152.stdout: auth debug = true 2024-09-12T06:27:39.672 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:27:39.672 INFO:teuthology.orchestra.run.smithi152.stdout: ms die on old message = true 2024-09-12T06:27:39.673 INFO:teuthology.orchestra.run.smithi152.stdout: ms die on bug = true 2024-09-12T06:27:39.673 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:27:39.673 INFO:teuthology.orchestra.run.smithi152.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-12T06:27:39.673 INFO:teuthology.orchestra.run.smithi152.stdout: mon pg warn max object skew = 0 2024-09-12T06:27:39.673 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:27:39.673 INFO:teuthology.orchestra.run.smithi152.stdout: # disable pg_autoscaler by default for new pools 2024-09-12T06:27:39.673 INFO:teuthology.orchestra.run.smithi152.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-12T06:27:39.673 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:27:39.673 INFO:teuthology.orchestra.run.smithi152.stdout: osd pool default size = 2 2024-09-12T06:27:39.673 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:27:39.673 INFO:teuthology.orchestra.run.smithi152.stdout: mon osd allow primary affinity = true 2024-09-12T06:27:39.673 INFO:teuthology.orchestra.run.smithi152.stdout: mon osd allow pg remap = true 2024-09-12T06:27:39.673 INFO:teuthology.orchestra.run.smithi152.stdout: mon warn on legacy crush tunables = false 2024-09-12T06:27:39.673 INFO:teuthology.orchestra.run.smithi152.stdout: mon warn on crush straw calc version zero = false 2024-09-12T06:27:39.673 INFO:teuthology.orchestra.run.smithi152.stdout: mon warn on no sortbitwise = false 2024-09-12T06:27:39.674 INFO:teuthology.orchestra.run.smithi152.stdout: mon warn on osd down out interval zero = false 2024-09-12T06:27:39.674 INFO:teuthology.orchestra.run.smithi152.stdout: mon warn on too few osds = false 2024-09-12T06:27:39.674 INFO:teuthology.orchestra.run.smithi152.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-12T06:27:39.674 INFO:teuthology.orchestra.run.smithi152.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-12T06:27:39.674 INFO:teuthology.orchestra.run.smithi152.stdout: mon_allow_pool_size_one = true 2024-09-12T06:27:39.674 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:27:39.674 INFO:teuthology.orchestra.run.smithi152.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-12T06:27:39.674 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:27:39.674 INFO:teuthology.orchestra.run.smithi152.stdout: osd default data pool replay window = 5 2024-09-12T06:27:39.674 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:27:39.674 INFO:teuthology.orchestra.run.smithi152.stdout: mon allow pool delete = true 2024-09-12T06:27:39.674 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:27:39.674 INFO:teuthology.orchestra.run.smithi152.stdout: mon cluster log file level = debug 2024-09-12T06:27:39.674 INFO:teuthology.orchestra.run.smithi152.stdout: debug asserts on shutdown = true 2024-09-12T06:27:39.674 INFO:teuthology.orchestra.run.smithi152.stdout: mon health detail to clog = false 2024-09-12T06:27:39.675 INFO:teuthology.orchestra.run.smithi152.stdout: mon host = 172.21.15.152 2024-09-12T06:27:39.675 INFO:teuthology.orchestra.run.smithi152.stdout: mon client directed command retry = 5 2024-09-12T06:27:39.675 INFO:teuthology.orchestra.run.smithi152.stdout: mon warn on pool no app = False 2024-09-12T06:27:39.675 INFO:teuthology.orchestra.run.smithi152.stdout: ms inject socket failures = 5000 2024-09-12T06:27:39.675 INFO:teuthology.orchestra.run.smithi152.stdout: fsid = cccfff5c-d879-49a4-9cf1-b06db11d0b7e 2024-09-12T06:27:39.675 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:27:39.675 INFO:teuthology.orchestra.run.smithi152.stdout:[osd] 2024-09-12T06:27:39.675 INFO:teuthology.orchestra.run.smithi152.stdout: osd journal size = 100 2024-09-12T06:27:39.675 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:27:39.675 INFO:teuthology.orchestra.run.smithi152.stdout: osd scrub load threshold = 5.0 2024-09-12T06:27:39.675 INFO:teuthology.orchestra.run.smithi152.stdout: osd scrub max interval = 600 2024-09-12T06:27:39.675 INFO:teuthology.orchestra.run.smithi152.stdout: osd mclock profile = high_recovery_ops 2024-09-12T06:27:39.675 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:27:39.675 INFO:teuthology.orchestra.run.smithi152.stdout: osd recover clone overlap = true 2024-09-12T06:27:39.675 INFO:teuthology.orchestra.run.smithi152.stdout: osd recovery max chunk = 1048576 2024-09-12T06:27:39.675 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:27:39.676 INFO:teuthology.orchestra.run.smithi152.stdout: osd debug shutdown = true 2024-09-12T06:27:39.676 INFO:teuthology.orchestra.run.smithi152.stdout: osd debug op order = true 2024-09-12T06:27:39.676 INFO:teuthology.orchestra.run.smithi152.stdout: osd debug verify stray on activate = true 2024-09-12T06:27:39.676 INFO:teuthology.orchestra.run.smithi152.stdout: osd debug trim objects = true 2024-09-12T06:27:39.676 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:27:39.676 INFO:teuthology.orchestra.run.smithi152.stdout: osd open classes on start = true 2024-09-12T06:27:39.676 INFO:teuthology.orchestra.run.smithi152.stdout: osd debug pg log writeout = true 2024-09-12T06:27:39.676 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:27:39.676 INFO:teuthology.orchestra.run.smithi152.stdout: osd deep scrub update digest min age = 30 2024-09-12T06:27:39.676 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:27:39.676 INFO:teuthology.orchestra.run.smithi152.stdout: osd map max advance = 10 2024-09-12T06:27:39.676 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:27:39.676 INFO:teuthology.orchestra.run.smithi152.stdout: journal zero on create = true 2024-09-12T06:27:39.677 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:27:39.677 INFO:teuthology.orchestra.run.smithi152.stdout: filestore ondisk finisher threads = 3 2024-09-12T06:27:39.677 INFO:teuthology.orchestra.run.smithi152.stdout: filestore apply finisher threads = 3 2024-09-12T06:27:39.677 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:27:39.677 INFO:teuthology.orchestra.run.smithi152.stdout: bdev debug aio = true 2024-09-12T06:27:39.677 INFO:teuthology.orchestra.run.smithi152.stdout: osd debug misdirected ops = true 2024-09-12T06:27:39.677 INFO:teuthology.orchestra.run.smithi152.stdout: bluestore block size = 96636764160 2024-09-12T06:27:39.677 INFO:teuthology.orchestra.run.smithi152.stdout: bluestore compression algorithm = lz4 2024-09-12T06:27:39.677 INFO:teuthology.orchestra.run.smithi152.stdout: bluestore compression mode = aggressive 2024-09-12T06:27:39.677 INFO:teuthology.orchestra.run.smithi152.stdout: bluestore fsck on mount = True 2024-09-12T06:27:39.677 INFO:teuthology.orchestra.run.smithi152.stdout: debug bluefs = 1/20 2024-09-12T06:27:39.677 INFO:teuthology.orchestra.run.smithi152.stdout: debug bluestore = 1/20 2024-09-12T06:27:39.677 INFO:teuthology.orchestra.run.smithi152.stdout: debug ms = 1 2024-09-12T06:27:39.677 INFO:teuthology.orchestra.run.smithi152.stdout: debug osd = 20 2024-09-12T06:27:39.677 INFO:teuthology.orchestra.run.smithi152.stdout: debug rocksdb = 4/10 2024-09-12T06:27:39.678 INFO:teuthology.orchestra.run.smithi152.stdout: mon osd backfillfull_ratio = 0.85 2024-09-12T06:27:39.678 INFO:teuthology.orchestra.run.smithi152.stdout: mon osd full ratio = 0.9 2024-09-12T06:27:39.678 INFO:teuthology.orchestra.run.smithi152.stdout: mon osd nearfull ratio = 0.8 2024-09-12T06:27:39.678 INFO:teuthology.orchestra.run.smithi152.stdout: osd failsafe full ratio = 0.95 2024-09-12T06:27:39.678 INFO:teuthology.orchestra.run.smithi152.stdout: osd objectstore = bluestore 2024-09-12T06:27:39.678 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:27:39.678 INFO:teuthology.orchestra.run.smithi152.stdout:[mgr] 2024-09-12T06:27:39.678 INFO:teuthology.orchestra.run.smithi152.stdout: debug ms = 1 2024-09-12T06:27:39.678 INFO:teuthology.orchestra.run.smithi152.stdout: debug mgr = 20 2024-09-12T06:27:39.678 INFO:teuthology.orchestra.run.smithi152.stdout: debug mon = 20 2024-09-12T06:27:39.678 INFO:teuthology.orchestra.run.smithi152.stdout: debug auth = 20 2024-09-12T06:27:39.678 INFO:teuthology.orchestra.run.smithi152.stdout: mon reweight min pgs per osd = 4 2024-09-12T06:27:39.678 INFO:teuthology.orchestra.run.smithi152.stdout: mon reweight min bytes per osd = 10 2024-09-12T06:27:39.678 INFO:teuthology.orchestra.run.smithi152.stdout: mgr/telemetry/nag = false 2024-09-12T06:27:39.678 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:27:39.679 INFO:teuthology.orchestra.run.smithi152.stdout:[mon] 2024-09-12T06:27:39.679 INFO:teuthology.orchestra.run.smithi152.stdout: debug ms = 1 2024-09-12T06:27:39.679 INFO:teuthology.orchestra.run.smithi152.stdout: debug mon = 20 2024-09-12T06:27:39.679 INFO:teuthology.orchestra.run.smithi152.stdout: debug paxos = 20 2024-09-12T06:27:39.679 INFO:teuthology.orchestra.run.smithi152.stdout: debug auth = 20 2024-09-12T06:27:39.679 INFO:teuthology.orchestra.run.smithi152.stdout: mon data avail warn = 5 2024-09-12T06:27:39.679 INFO:teuthology.orchestra.run.smithi152.stdout: mon mgr mkfs grace = 240 2024-09-12T06:27:39.679 INFO:teuthology.orchestra.run.smithi152.stdout: mon reweight min pgs per osd = 4 2024-09-12T06:27:39.679 INFO:teuthology.orchestra.run.smithi152.stdout: mon osd reporter subtree level = osd 2024-09-12T06:27:39.679 INFO:teuthology.orchestra.run.smithi152.stdout: mon osd prime pg temp = true 2024-09-12T06:27:39.679 INFO:teuthology.orchestra.run.smithi152.stdout: mon reweight min bytes per osd = 10 2024-09-12T06:27:39.679 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:27:39.679 INFO:teuthology.orchestra.run.smithi152.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-12T06:27:39.679 INFO:teuthology.orchestra.run.smithi152.stdout: auth mon ticket ttl = 660 # 11m 2024-09-12T06:27:39.679 INFO:teuthology.orchestra.run.smithi152.stdout: auth service ticket ttl = 240 # 4m 2024-09-12T06:27:39.680 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:27:39.680 INFO:teuthology.orchestra.run.smithi152.stdout: # don't complain about insecure global_id in the test suite 2024-09-12T06:27:39.680 INFO:teuthology.orchestra.run.smithi152.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-12T06:27:39.680 INFO:teuthology.orchestra.run.smithi152.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-12T06:27:39.680 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:27:39.680 INFO:teuthology.orchestra.run.smithi152.stdout: # 1m isn't quite enough 2024-09-12T06:27:39.680 INFO:teuthology.orchestra.run.smithi152.stdout: mon_down_mkfs_grace = 2m 2024-09-12T06:27:39.680 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:27:39.680 INFO:teuthology.orchestra.run.smithi152.stdout: mon_warn_on_filestore_osds = false 2024-09-12T06:27:39.680 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:27:39.680 INFO:teuthology.orchestra.run.smithi152.stdout:[client] 2024-09-12T06:27:39.680 INFO:teuthology.orchestra.run.smithi152.stdout: rgw cache enabled = true 2024-09-12T06:27:39.680 INFO:teuthology.orchestra.run.smithi152.stdout: rgw enable ops log = true 2024-09-12T06:27:39.680 INFO:teuthology.orchestra.run.smithi152.stdout: rgw enable usage log = true 2024-09-12T06:27:39.680 INFO:teuthology.orchestra.run.smithi152.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-12T06:27:39.680 INFO:teuthology.orchestra.run.smithi152.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-12T06:27:39.681 INFO:teuthology.orchestra.run.smithi152.stdout: debug rbd = 20 2024-09-12T06:27:39.681 INFO:teuthology.orchestra.run.smithi152.stdout: debug rbd_mirror = 15 2024-09-12T06:27:39.681 INFO:teuthology.orchestra.run.smithi152.stdout: log to stderr = False 2024-09-12T06:27:39.681 INFO:teuthology.orchestra.run.smithi152.stdout: rbd default features = 125 2024-09-12T06:27:39.681 INFO:teuthology.orchestra.run.smithi152.stdout:[mon.a] 2024-09-12T06:27:39.681 INFO:teuthology.orchestra.run.smithi152.stdout:[client.mirror.0] 2024-09-12T06:27:39.681 INFO:teuthology.orchestra.run.smithi152.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-12T06:27:39.681 INFO:teuthology.orchestra.run.smithi152.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-12T06:27:39.681 INFO:teuthology.orchestra.run.smithi152.stdout:[client.mirror.1] 2024-09-12T06:27:39.681 INFO:teuthology.orchestra.run.smithi152.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-12T06:27:39.681 INFO:teuthology.orchestra.run.smithi152.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-12T06:27:39.681 INFO:teuthology.orchestra.run.smithi152.stdout:[client.mirror.2] 2024-09-12T06:27:39.681 INFO:teuthology.orchestra.run.smithi152.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-12T06:27:39.681 INFO:teuthology.orchestra.run.smithi152.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-12T06:27:39.681 INFO:teuthology.orchestra.run.smithi152.stdout:[client.mirror.3] 2024-09-12T06:27:39.681 INFO:teuthology.orchestra.run.smithi152.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-12T06:27:39.682 INFO:teuthology.orchestra.run.smithi152.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-12T06:27:39.682 INFO:teuthology.orchestra.run.smithi152.stdout:[client.mirror.4] 2024-09-12T06:27:39.682 INFO:teuthology.orchestra.run.smithi152.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-12T06:27:39.682 INFO:teuthology.orchestra.run.smithi152.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-12T06:27:39.682 INFO:teuthology.orchestra.run.smithi152.stdout:[client.mirror.5] 2024-09-12T06:27:39.682 INFO:teuthology.orchestra.run.smithi152.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-12T06:27:39.682 INFO:teuthology.orchestra.run.smithi152.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-12T06:27:39.682 INFO:teuthology.orchestra.run.smithi152.stdout:[client.mirror.6] 2024-09-12T06:27:39.682 INFO:teuthology.orchestra.run.smithi152.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-12T06:27:39.682 INFO:teuthology.orchestra.run.smithi152.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-12T06:27:39.683 INFO:tasks.ceph:Creating admin key on cluster2.mon.a... 2024-09-12T06:27:39.683 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster2.keyring 2024-09-12T06:27:39.777 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-12T06:27:39.777 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-12T06:27:39.777 DEBUG:teuthology.orchestra.run.smithi152:> dd if=/etc/ceph/cluster2.keyring of=/dev/stdout 2024-09-12T06:27:39.793 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-12T06:27:39.793 DEBUG:teuthology.orchestra.run.smithi152:> dd if=/home/ubuntu/cephtest/cluster2.monmap of=/dev/stdout 2024-09-12T06:27:39.848 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi096.front.sepia.ceph.com 2024-09-12T06:27:39.848 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-12T06:27:39.848 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/etc/ceph/cluster2.keyring 2024-09-12T06:27:39.848 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-09-12T06:27:39.886 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-12T06:27:39.886 DEBUG:teuthology.orchestra.run.smithi096:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2024-09-12T06:27:39.943 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi152.front.sepia.ceph.com 2024-09-12T06:27:39.944 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-12T06:27:39.944 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/etc/ceph/cluster2.keyring 2024-09-12T06:27:39.944 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-09-12T06:27:39.977 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-12T06:27:39.978 DEBUG:teuthology.orchestra.run.smithi152:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2024-09-12T06:27:40.033 INFO:tasks.ceph:Setting up mon nodes... 2024-09-12T06:27:40.033 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-12T06:27:40.033 DEBUG:teuthology.orchestra.run.smithi096:> sudo mkdir -p /var/lib/ceph/mgr/cluster2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster2-x/keyring 2024-09-12T06:27:40.086 INFO:teuthology.orchestra.run.smithi096.stdout:creating /var/lib/ceph/mgr/cluster2-x/keyring 2024-09-12T06:27:40.089 INFO:tasks.ceph:Setting up mds nodes... 2024-09-12T06:27:40.089 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-12T06:27:40.089 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster2.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.0.keyring 2024-09-12T06:27:40.131 INFO:teuthology.orchestra.run.smithi096.stdout:creating /etc/ceph/cluster2.client.0.keyring 2024-09-12T06:27:40.144 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster2.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.keyring 2024-09-12T06:27:40.185 INFO:teuthology.orchestra.run.smithi152.stdout:creating /etc/ceph/cluster2.client.mirror.keyring 2024-09-12T06:27:40.198 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster2.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.0.keyring 2024-09-12T06:27:40.278 INFO:teuthology.orchestra.run.smithi152.stdout:creating /etc/ceph/cluster2.client.mirror.0.keyring 2024-09-12T06:27:40.291 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster2.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.1.keyring 2024-09-12T06:27:40.374 INFO:teuthology.orchestra.run.smithi152.stdout:creating /etc/ceph/cluster2.client.mirror.1.keyring 2024-09-12T06:27:40.386 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster2.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.2.keyring 2024-09-12T06:27:40.468 INFO:teuthology.orchestra.run.smithi152.stdout:creating /etc/ceph/cluster2.client.mirror.2.keyring 2024-09-12T06:27:40.481 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster2.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.3.keyring 2024-09-12T06:27:40.562 INFO:teuthology.orchestra.run.smithi152.stdout:creating /etc/ceph/cluster2.client.mirror.3.keyring 2024-09-12T06:27:40.575 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster2.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.4.keyring 2024-09-12T06:27:40.656 INFO:teuthology.orchestra.run.smithi152.stdout:creating /etc/ceph/cluster2.client.mirror.4.keyring 2024-09-12T06:27:40.669 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster2.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.5.keyring 2024-09-12T06:27:40.750 INFO:teuthology.orchestra.run.smithi152.stdout:creating /etc/ceph/cluster2.client.mirror.5.keyring 2024-09-12T06:27:40.763 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster2.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.6.keyring 2024-09-12T06:27:40.844 INFO:teuthology.orchestra.run.smithi152.stdout:creating /etc/ceph/cluster2.client.mirror.6.keyring 2024-09-12T06:27:40.857 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-12T06:27:40.857 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi096.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi152.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-12T06:27:40.857 DEBUG:teuthology.orchestra.run.smithi152:> sudo mkdir -p /var/lib/ceph/osd/cluster2-0 2024-09-12T06:27:40.921 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-12T06:27:40.921 INFO:tasks.ceph:role: cluster2.osd.0 2024-09-12T06:27:40.921 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi152.front.sepia.ceph.com 2024-09-12T06:27:40.922 DEBUG:teuthology.orchestra.run.smithi152:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-12T06:27:40.994 INFO:teuthology.orchestra.run.smithi152.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-12T06:27:40.994 INFO:teuthology.orchestra.run.smithi152.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-12T06:27:40.994 INFO:teuthology.orchestra.run.smithi152.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-12T06:27:40.994 INFO:teuthology.orchestra.run.smithi152.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-12T06:27:40.994 INFO:teuthology.orchestra.run.smithi152.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-12T06:27:40.994 INFO:teuthology.orchestra.run.smithi152.stdout: = sunit=0 swidth=0 blks 2024-09-12T06:27:40.994 INFO:teuthology.orchestra.run.smithi152.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-12T06:27:40.995 INFO:teuthology.orchestra.run.smithi152.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-12T06:27:40.995 INFO:teuthology.orchestra.run.smithi152.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-12T06:27:40.995 INFO:teuthology.orchestra.run.smithi152.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-12T06:27:41.508 INFO:teuthology.orchestra.run.smithi152.stdout:Discarding blocks...Done. 2024-09-12T06:27:41.510 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi152.front.sepia.ceph.com -o noatime 2024-09-12T06:27:41.511 DEBUG:teuthology.orchestra.run.smithi152:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster2-0 2024-09-12T06:27:41.650 DEBUG:teuthology.orchestra.run.smithi152:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-0 2024-09-12T06:27:41.683 DEBUG:teuthology.orchestra.run.smithi152:> sudo mkdir -p /var/lib/ceph/osd/cluster2-1 2024-09-12T06:27:41.748 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-12T06:27:41.748 INFO:tasks.ceph:role: cluster2.osd.1 2024-09-12T06:27:41.749 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi152.front.sepia.ceph.com 2024-09-12T06:27:41.749 DEBUG:teuthology.orchestra.run.smithi152:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-12T06:27:41.821 INFO:teuthology.orchestra.run.smithi152.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-12T06:27:41.821 INFO:teuthology.orchestra.run.smithi152.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-12T06:27:41.821 INFO:teuthology.orchestra.run.smithi152.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-12T06:27:41.821 INFO:teuthology.orchestra.run.smithi152.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-12T06:27:41.821 INFO:teuthology.orchestra.run.smithi152.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-12T06:27:41.821 INFO:teuthology.orchestra.run.smithi152.stdout: = sunit=0 swidth=0 blks 2024-09-12T06:27:41.821 INFO:teuthology.orchestra.run.smithi152.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-12T06:27:41.821 INFO:teuthology.orchestra.run.smithi152.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-12T06:27:41.821 INFO:teuthology.orchestra.run.smithi152.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-12T06:27:41.821 INFO:teuthology.orchestra.run.smithi152.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-12T06:27:42.272 INFO:teuthology.orchestra.run.smithi152.stdout:Discarding blocks...Done. 2024-09-12T06:27:42.274 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi152.front.sepia.ceph.com -o noatime 2024-09-12T06:27:42.274 DEBUG:teuthology.orchestra.run.smithi152:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster2-1 2024-09-12T06:27:42.355 DEBUG:teuthology.orchestra.run.smithi152:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-1 2024-09-12T06:27:42.425 DEBUG:teuthology.orchestra.run.smithi152:> sudo mkdir -p /var/lib/ceph/osd/cluster2-2 2024-09-12T06:27:42.491 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-12T06:27:42.491 INFO:tasks.ceph:role: cluster2.osd.2 2024-09-12T06:27:42.491 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi152.front.sepia.ceph.com 2024-09-12T06:27:42.492 DEBUG:teuthology.orchestra.run.smithi152:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-12T06:27:42.564 INFO:teuthology.orchestra.run.smithi152.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-12T06:27:42.564 INFO:teuthology.orchestra.run.smithi152.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-12T06:27:42.564 INFO:teuthology.orchestra.run.smithi152.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-12T06:27:42.564 INFO:teuthology.orchestra.run.smithi152.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-12T06:27:42.564 INFO:teuthology.orchestra.run.smithi152.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-12T06:27:42.564 INFO:teuthology.orchestra.run.smithi152.stdout: = sunit=0 swidth=0 blks 2024-09-12T06:27:42.565 INFO:teuthology.orchestra.run.smithi152.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-12T06:27:42.565 INFO:teuthology.orchestra.run.smithi152.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-12T06:27:42.565 INFO:teuthology.orchestra.run.smithi152.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-12T06:27:42.565 INFO:teuthology.orchestra.run.smithi152.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-12T06:27:43.051 INFO:teuthology.orchestra.run.smithi152.stdout:Discarding blocks...Done. 2024-09-12T06:27:43.053 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi152.front.sepia.ceph.com -o noatime 2024-09-12T06:27:43.053 DEBUG:teuthology.orchestra.run.smithi152:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster2-2 2024-09-12T06:27:43.183 DEBUG:teuthology.orchestra.run.smithi152:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-2 2024-09-12T06:27:43.214 DEBUG:teuthology.orchestra.run.smithi152:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-09-12T06:27:43.312 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-12T06:27:43.314+0000 7f22d67fd640 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-0/keyring: can't open /var/lib/ceph/osd/cluster2-0/keyring: (2) No such file or directory 2024-09-12T06:27:43.317 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-12T06:27:43.319+0000 7f22d67fd640 -1 created new key in keyring /var/lib/ceph/osd/cluster2-0/keyring 2024-09-12T06:27:43.317 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-12T06:27:43.319+0000 7f22d67fd640 -1 bdev(0x55c37a04b800 /var/lib/ceph/osd/cluster2-0/block) open stat got: (1) Operation not permitted 2024-09-12T06:27:43.317 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-12T06:27:43.319+0000 7f22d67fd640 -1 bluestore(/var/lib/ceph/osd/cluster2-0) _read_fsid unparsable uuid 2024-09-12T06:27:46.479 DEBUG:teuthology.orchestra.run.smithi152:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-0 2024-09-12T06:27:46.506 DEBUG:teuthology.orchestra.run.smithi152:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-09-12T06:27:46.595 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-12T06:27:46.597+0000 7f48acb34640 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-1/keyring: can't open /var/lib/ceph/osd/cluster2-1/keyring: (2) No such file or directory 2024-09-12T06:27:46.595 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-12T06:27:46.597+0000 7f48acb34640 -1 created new key in keyring /var/lib/ceph/osd/cluster2-1/keyring 2024-09-12T06:27:46.595 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-12T06:27:46.597+0000 7f48acb34640 -1 bdev(0x5605c3fbf800 /var/lib/ceph/osd/cluster2-1/block) open stat got: (1) Operation not permitted 2024-09-12T06:27:46.595 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-12T06:27:46.597+0000 7f48acb34640 -1 bluestore(/var/lib/ceph/osd/cluster2-1) _read_fsid unparsable uuid 2024-09-12T06:27:49.498 DEBUG:teuthology.orchestra.run.smithi152:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-1 2024-09-12T06:27:49.525 DEBUG:teuthology.orchestra.run.smithi152:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-09-12T06:27:49.613 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-12T06:27:49.614+0000 7fdce220b640 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-2/keyring: can't open /var/lib/ceph/osd/cluster2-2/keyring: (2) No such file or directory 2024-09-12T06:27:49.613 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-12T06:27:49.615+0000 7fdce220b640 -1 created new key in keyring /var/lib/ceph/osd/cluster2-2/keyring 2024-09-12T06:27:49.613 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-12T06:27:49.615+0000 7fdce220b640 -1 bdev(0x5585e2f21800 /var/lib/ceph/osd/cluster2-2/block) open stat got: (1) Operation not permitted 2024-09-12T06:27:49.613 INFO:teuthology.orchestra.run.smithi152.stderr:2024-09-12T06:27:49.615+0000 7fdce220b640 -1 bluestore(/var/lib/ceph/osd/cluster2-2) _read_fsid unparsable uuid 2024-09-12T06:27:53.017 DEBUG:teuthology.orchestra.run.smithi152:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-2 2024-09-12T06:27:53.046 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-12T06:27:53.046 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-12T06:27:53.046 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/var/lib/ceph/mgr/cluster2-x/keyring of=/dev/stdout 2024-09-12T06:27:53.077 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-12T06:27:53.077 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/var/lib/ceph/osd/cluster2-0/keyring of=/dev/stdout 2024-09-12T06:27:53.113 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-12T06:27:53.113 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/var/lib/ceph/osd/cluster2-1/keyring of=/dev/stdout 2024-09-12T06:27:53.180 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-12T06:27:53.180 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/var/lib/ceph/osd/cluster2-2/keyring of=/dev/stdout 2024-09-12T06:27:53.247 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-12T06:27:53.247 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/etc/ceph/cluster2.client.0.keyring of=/dev/stdout 2024-09-12T06:27:53.266 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-12T06:27:53.266 DEBUG:teuthology.orchestra.run.smithi152:> dd if=/etc/ceph/cluster2.client.mirror.keyring of=/dev/stdout 2024-09-12T06:27:53.303 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-12T06:27:53.303 DEBUG:teuthology.orchestra.run.smithi152:> dd if=/etc/ceph/cluster2.client.mirror.0.keyring of=/dev/stdout 2024-09-12T06:27:53.359 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-12T06:27:53.359 DEBUG:teuthology.orchestra.run.smithi152:> dd if=/etc/ceph/cluster2.client.mirror.1.keyring of=/dev/stdout 2024-09-12T06:27:53.414 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-12T06:27:53.415 DEBUG:teuthology.orchestra.run.smithi152:> dd if=/etc/ceph/cluster2.client.mirror.2.keyring of=/dev/stdout 2024-09-12T06:27:53.470 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-12T06:27:53.471 DEBUG:teuthology.orchestra.run.smithi152:> dd if=/etc/ceph/cluster2.client.mirror.3.keyring of=/dev/stdout 2024-09-12T06:27:53.526 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-12T06:27:53.526 DEBUG:teuthology.orchestra.run.smithi152:> dd if=/etc/ceph/cluster2.client.mirror.4.keyring of=/dev/stdout 2024-09-12T06:27:53.581 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-12T06:27:53.581 DEBUG:teuthology.orchestra.run.smithi152:> dd if=/etc/ceph/cluster2.client.mirror.5.keyring of=/dev/stdout 2024-09-12T06:27:53.636 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-12T06:27:53.636 DEBUG:teuthology.orchestra.run.smithi152:> dd if=/etc/ceph/cluster2.client.mirror.6.keyring of=/dev/stdout 2024-09-12T06:27:53.690 INFO:tasks.ceph:Adding keys to all mons... 2024-09-12T06:27:53.691 DEBUG:teuthology.orchestra.run.smithi152:> sudo tee -a /etc/ceph/cluster2.keyring 2024-09-12T06:27:53.754 INFO:teuthology.orchestra.run.smithi152.stdout:[mgr.x] 2024-09-12T06:27:53.754 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQDcieJm7qlfBRAAijoQ1ECaPGESRvkjw8Lqlw== 2024-09-12T06:27:53.754 INFO:teuthology.orchestra.run.smithi152.stdout:[osd.0] 2024-09-12T06:27:53.754 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQDfieJm/o3REhAA5vmz7t/YkZtIHiBgRJnZ7Q== 2024-09-12T06:27:53.754 INFO:teuthology.orchestra.run.smithi152.stdout:[osd.1] 2024-09-12T06:27:53.754 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQDiieJmUsypIxAAlV4NuiTOtvw7qmPGw3i6zw== 2024-09-12T06:27:53.754 INFO:teuthology.orchestra.run.smithi152.stdout:[osd.2] 2024-09-12T06:27:53.754 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQDlieJmpdW0JBAA2BLO5savxf2Xjm79xDP7fA== 2024-09-12T06:27:53.754 INFO:teuthology.orchestra.run.smithi152.stdout:[client.0] 2024-09-12T06:27:53.754 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQDcieJmfKUACBAAoStysbYNB6jVsbkTrF1X+w== 2024-09-12T06:27:53.754 INFO:teuthology.orchestra.run.smithi152.stdout:[client.mirror] 2024-09-12T06:27:53.754 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQDcieJmk2g8CxAA/AalP6wEGk/5Cqwj7LaupA== 2024-09-12T06:27:53.755 INFO:teuthology.orchestra.run.smithi152.stdout:[client.mirror.0] 2024-09-12T06:27:53.755 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQDcieJm13rOEBAAv8xtt/WFjSnnZ995dHCWKA== 2024-09-12T06:27:53.755 INFO:teuthology.orchestra.run.smithi152.stdout:[client.mirror.1] 2024-09-12T06:27:53.755 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQDcieJmf5V9FhAAkzMzgAWfzmIhg/c6Frz0KQ== 2024-09-12T06:27:53.755 INFO:teuthology.orchestra.run.smithi152.stdout:[client.mirror.2] 2024-09-12T06:27:53.755 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQDcieJmNEIjHBAArkTZWYIeWwB23WCp3SW0eg== 2024-09-12T06:27:53.755 INFO:teuthology.orchestra.run.smithi152.stdout:[client.mirror.3] 2024-09-12T06:27:53.755 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQDcieJmyoe9IRAAV3vMCSa6ylnN0VX3S++mrg== 2024-09-12T06:27:53.755 INFO:teuthology.orchestra.run.smithi152.stdout:[client.mirror.4] 2024-09-12T06:27:53.755 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQDcieJmEF5TJxAAdSF4dV0ZbPS+PY4h9zVTww== 2024-09-12T06:27:53.755 INFO:teuthology.orchestra.run.smithi152.stdout:[client.mirror.5] 2024-09-12T06:27:53.755 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQDcieJmx1noLBAAKJKtbgmDvr6YEuTli5zKpw== 2024-09-12T06:27:53.755 INFO:teuthology.orchestra.run.smithi152.stdout:[client.mirror.6] 2024-09-12T06:27:53.755 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQDcieJmc8SBMhAAy603eaRrC05q4mPHom0PYg== 2024-09-12T06:27:53.756 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-12T06:27:53.848 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-12T06:27:53.899 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-12T06:27:53.949 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-12T06:27:54.000 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-12T06:27:54.054 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-12T06:27:54.108 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-12T06:27:54.162 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-12T06:27:54.215 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-12T06:27:54.269 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-12T06:27:54.322 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-12T06:27:54.376 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-12T06:27:54.431 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-12T06:27:54.485 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-12T06:27:54.485 DEBUG:teuthology.orchestra.run.smithi152:> sudo mkdir -p /var/lib/ceph/mon/cluster2-a 2024-09-12T06:27:54.510 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster2 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster2.monmap --keyring /etc/ceph/cluster2.keyring 2024-09-12T06:27:54.605 DEBUG:teuthology.orchestra.run.smithi152:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster2-a 2024-09-12T06:27:54.634 DEBUG:teuthology.orchestra.run.smithi152:> rm -- /home/ubuntu/cephtest/cluster2.monmap 2024-09-12T06:27:54.689 INFO:tasks.ceph:Starting mon daemons in cluster cluster2... 2024-09-12T06:27:54.689 INFO:tasks.ceph.cluster2.mon.a:Restarting daemon 2024-09-12T06:27:54.689 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster2 -i a 2024-09-12T06:27:54.731 INFO:tasks.ceph.cluster2.mon.a:Started 2024-09-12T06:27:54.731 INFO:tasks.ceph:Starting mgr daemons in cluster cluster2... 2024-09-12T06:27:54.731 INFO:tasks.ceph.cluster2.mgr.x:Restarting daemon 2024-09-12T06:27:54.732 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster2 -i x 2024-09-12T06:27:54.734 INFO:tasks.ceph.cluster2.mgr.x:Started 2024-09-12T06:27:54.734 DEBUG:tasks.ceph:set 0 configs 2024-09-12T06:27:54.735 INFO:tasks.ceph:Setting crush tunables to default 2024-09-12T06:27:54.735 DEBUG:teuthology.orchestra.run.smithi152:> sudo ceph --cluster cluster2 osd crush tunables default 2024-09-12T06:27:54.904 INFO:teuthology.orchestra.run.smithi152.stderr:adjusted tunables profile to default 2024-09-12T06:27:54.915 INFO:tasks.ceph:check_enable_crimson: False 2024-09-12T06:27:54.915 INFO:tasks.ceph:Starting osd daemons in cluster cluster2... 2024-09-12T06:27:54.915 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-12T06:27:54.915 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/var/lib/ceph/osd/cluster2-0/fsid of=/dev/stdout 2024-09-12T06:27:54.940 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-12T06:27:54.940 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/var/lib/ceph/osd/cluster2-1/fsid of=/dev/stdout 2024-09-12T06:27:55.004 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-12T06:27:55.004 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/var/lib/ceph/osd/cluster2-2/fsid of=/dev/stdout 2024-09-12T06:27:55.068 DEBUG:teuthology.orchestra.run.smithi152:> sudo ceph --cluster cluster2 osd new cdacbd5a-efce-4370-b990-b1e75740a4cc 0 2024-09-12T06:27:55.283 INFO:teuthology.orchestra.run.smithi152.stdout:0 2024-09-12T06:27:55.294 DEBUG:teuthology.orchestra.run.smithi152:> sudo ceph --cluster cluster2 osd new 9019ee56-f19d-443f-99a7-7e04ef9915b7 1 2024-09-12T06:27:55.468 INFO:teuthology.orchestra.run.smithi152.stdout:1 2024-09-12T06:27:55.479 DEBUG:teuthology.orchestra.run.smithi152:> sudo ceph --cluster cluster2 osd new 47b01d4d-e415-4491-a522-9513d1569287 2 2024-09-12T06:27:55.654 INFO:teuthology.orchestra.run.smithi152.stdout:2 2024-09-12T06:27:55.665 INFO:tasks.ceph.cluster2.osd.0:Restarting daemon 2024-09-12T06:27:55.665 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 0 2024-09-12T06:27:55.667 INFO:tasks.ceph.cluster2.osd.0:Started 2024-09-12T06:27:55.668 INFO:tasks.ceph.cluster2.osd.1:Restarting daemon 2024-09-12T06:27:55.668 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 1 2024-09-12T06:27:55.670 INFO:tasks.ceph.cluster2.osd.1:Started 2024-09-12T06:27:55.670 INFO:tasks.ceph.cluster2.osd.2:Restarting daemon 2024-09-12T06:27:55.670 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 2 2024-09-12T06:27:55.672 INFO:tasks.ceph.cluster2.osd.2:Started 2024-09-12T06:27:55.672 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-09-12T06:27:55.843 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:27:55.843 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":5,"fsid":"cccfff5c-d879-49a4-9cf1-b06db11d0b7e","created":"2024-09-12T06:27:54.820385+0000","modified":"2024-09-12T06:27:55.656795+0000","last_up_change":"0.000000","last_in_change":"2024-09-12T06:27:55.656795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cdacbd5a-efce-4370-b990-b1e75740a4cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9019ee56-f19d-443f-99a7-7e04ef9915b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47b01d4d-e415-4491-a522-9513d1569287","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-12T06:27:55.854 INFO:tasks.ceph.ceph_manager.cluster2:[] 2024-09-12T06:27:55.855 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-12T06:27:56.157 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-09-12T06:27:56.316 INFO:teuthology.misc.health.smithi152.stdout: 2024-09-12T06:27:56.316 INFO:teuthology.misc.health.smithi152.stdout:{"epoch":5,"fsid":"cccfff5c-d879-49a4-9cf1-b06db11d0b7e","created":"2024-09-12T06:27:54.820385+0000","modified":"2024-09-12T06:27:55.656795+0000","last_up_change":"0.000000","last_in_change":"2024-09-12T06:27:55.656795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cdacbd5a-efce-4370-b990-b1e75740a4cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9019ee56-f19d-443f-99a7-7e04ef9915b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47b01d4d-e415-4491-a522-9513d1569287","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-12T06:27:56.325 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-12T06:27:56.859 INFO:tasks.ceph.cluster2.osd.2.smithi152.stderr:2024-09-12T06:27:56.861+0000 7f479aa0f640 -1 Falling back to public interface 2024-09-12T06:27:56.898 INFO:tasks.ceph.cluster2.osd.1.smithi152.stderr:2024-09-12T06:27:56.900+0000 7f340aa11640 -1 Falling back to public interface 2024-09-12T06:27:57.109 INFO:tasks.ceph.cluster2.osd.0.smithi152.stderr:2024-09-12T06:27:57.111+0000 7fd62761c640 -1 Falling back to public interface 2024-09-12T06:27:59.505 INFO:tasks.ceph.cluster2.mgr.x.smithi096.stderr:2024-09-12T06:27:59.507+0000 7f419d889640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-09-12T06:28:00.666 INFO:tasks.ceph.cluster2.osd.2.smithi152.stderr:2024-09-12T06:28:00.668+0000 7f479aa0f640 -1 osd.2 0 log_to_monitors true 2024-09-12T06:28:00.695 INFO:tasks.ceph.cluster2.osd.1.smithi152.stderr:2024-09-12T06:28:00.697+0000 7f340aa11640 -1 osd.1 0 log_to_monitors true 2024-09-12T06:28:00.878 INFO:tasks.ceph.cluster2.osd.0.smithi152.stderr:2024-09-12T06:28:00.880+0000 7fd62761c640 -1 osd.0 0 log_to_monitors true 2024-09-12T06:28:02.628 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-09-12T06:28:03.070 INFO:teuthology.misc.health.smithi152.stdout: 2024-09-12T06:28:03.070 INFO:teuthology.misc.health.smithi152.stdout:{"epoch":7,"fsid":"cccfff5c-d879-49a4-9cf1-b06db11d0b7e","created":"2024-09-12T06:27:54.820385+0000","modified":"2024-09-12T06:28:02.305578+0000","last_up_change":"0.000000","last_in_change":"2024-09-12T06:27:55.656795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cdacbd5a-efce-4370-b990-b1e75740a4cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9019ee56-f19d-443f-99a7-7e04ef9915b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47b01d4d-e415-4491-a522-9513d1569287","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-12T06:28:03.085 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-12T06:28:03.327 INFO:tasks.ceph.cluster2.osd.0.smithi152.stderr:2024-09-12T06:28:03.329+0000 7fd622fa1640 -1 osd.0 0 waiting for initial osdmap 2024-09-12T06:28:03.337 INFO:tasks.ceph.cluster2.osd.0.smithi152.stderr:2024-09-12T06:28:03.339+0000 7fd61adc0640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-12T06:28:03.354 INFO:tasks.ceph.cluster2.osd.1.smithi152.stderr:2024-09-12T06:28:03.356+0000 7f3406392640 -1 osd.1 0 waiting for initial osdmap 2024-09-12T06:28:03.363 INFO:tasks.ceph.cluster2.osd.1.smithi152.stderr:2024-09-12T06:28:03.365+0000 7f33fe1b1640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-12T06:28:03.373 INFO:tasks.ceph.cluster2.osd.2.smithi152.stderr:2024-09-12T06:28:03.375+0000 7f47962ed640 -1 osd.2 0 waiting for initial osdmap 2024-09-12T06:28:03.383 INFO:tasks.ceph.cluster2.osd.2.smithi152.stderr:2024-09-12T06:28:03.385+0000 7f478e10c640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-12T06:28:09.388 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-09-12T06:28:09.675 INFO:teuthology.misc.health.smithi152.stdout: 2024-09-12T06:28:09.675 INFO:teuthology.misc.health.smithi152.stdout:{"epoch":11,"fsid":"cccfff5c-d879-49a4-9cf1-b06db11d0b7e","created":"2024-09-12T06:27:54.820385+0000","modified":"2024-09-12T06:28:07.336384+0000","last_up_change":"2024-09-12T06:28:04.331593+0000","last_in_change":"2024-09-12T06:27:55.656795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-12T06:28:04.463680+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cdacbd5a-efce-4370-b990-b1e75740a4cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":2673152699},{"type":"v1","addr":"172.21.15.152:6817","nonce":2673152699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":2673152699},{"type":"v1","addr":"172.21.15.152:6819","nonce":2673152699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":2673152699},{"type":"v1","addr":"172.21.15.152:6823","nonce":2673152699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":2673152699},{"type":"v1","addr":"172.21.15.152:6821","nonce":2673152699}]},"public_addr":"172.21.15.152:6817/2673152699","cluster_addr":"172.21.15.152:6819/2673152699","heartbeat_back_addr":"172.21.15.152:6823/2673152699","heartbeat_front_addr":"172.21.15.152:6821/2673152699","state":["exists","up"]},{"osd":1,"uuid":"9019ee56-f19d-443f-99a7-7e04ef9915b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":1156225285},{"type":"v1","addr":"172.21.15.152:6809","nonce":1156225285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":1156225285},{"type":"v1","addr":"172.21.15.152:6811","nonce":1156225285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":1156225285},{"type":"v1","addr":"172.21.15.152:6815","nonce":1156225285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":1156225285},{"type":"v1","addr":"172.21.15.152:6813","nonce":1156225285}]},"public_addr":"172.21.15.152:6809/1156225285","cluster_addr":"172.21.15.152:6811/1156225285","heartbeat_back_addr":"172.21.15.152:6815/1156225285","heartbeat_front_addr":"172.21.15.152:6813/1156225285","state":["exists","up"]},{"osd":2,"uuid":"47b01d4d-e415-4491-a522-9513d1569287","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":2164790219},{"type":"v1","addr":"172.21.15.152:6801","nonce":2164790219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":2164790219},{"type":"v1","addr":"172.21.15.152:6803","nonce":2164790219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":2164790219},{"type":"v1","addr":"172.21.15.152:6807","nonce":2164790219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":2164790219},{"type":"v1","addr":"172.21.15.152:6805","nonce":2164790219}]},"public_addr":"172.21.15.152:6801/2164790219","cluster_addr":"172.21.15.152:6803/2164790219","heartbeat_back_addr":"172.21.15.152:6807/2164790219","heartbeat_front_addr":"172.21.15.152:6805/2164790219","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T06:28:01.930599+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T06:28:01.728162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T06:28:01.719714+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-12T06:28:09.687 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-09-12T06:28:09.687 INFO:tasks.ceph:Creating RBD pool 2024-09-12T06:28:09.687 DEBUG:teuthology.orchestra.run.smithi152:> sudo ceph --cluster cluster2 osd pool create rbd 8 2024-09-12T06:28:10.346 INFO:teuthology.orchestra.run.smithi152.stderr:pool 'rbd' created 2024-09-12T06:28:10.364 DEBUG:teuthology.orchestra.run.smithi152:> rbd --cluster cluster2 pool init rbd 2024-09-12T06:28:13.381 INFO:tasks.ceph:Starting mds daemons in cluster cluster2... 2024-09-12T06:28:13.382 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 config log 1 --format=json 2024-09-12T06:28:13.383 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-12T06:28:13.681 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:28:13.694 INFO:teuthology.orchestra.run.smithi152.stdout:[{"version":4,"timestamp":"2024-09-12T06:28:03.376858+0000","name":"","changes":[{"name":"osd.2/osd_mclock_max_capacity_iops_ssd","new_value":"4554.844585"}]}] 2024-09-12T06:28:13.694 INFO:tasks.ceph_manager:config epoch is 4 2024-09-12T06:28:13.695 INFO:tasks.ceph:Waiting until cluster2 daemons up and pgs clean... 2024-09-12T06:28:13.695 INFO:tasks.ceph.ceph_manager.cluster2:waiting for mgr available 2024-09-12T06:28:13.695 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 mgr dump --format=json 2024-09-12T06:28:14.026 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:28:14.037 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":5,"flags":0,"active_gid":4098,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":4095266092},{"type":"v1","addr":"172.21.15.96:6827","nonce":4095266092}]},"active_addr":"172.21.15.96:6827/4095266092","active_change":"2024-09-12T06:27:58.241986+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.96:0","nonce":486417921}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.96:0","nonce":1180396387}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.96:0","nonce":2156544470}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.96:0","nonce":4077703258}]}]} 2024-09-12T06:28:14.039 INFO:tasks.ceph.ceph_manager.cluster2:mgr available! 2024-09-12T06:28:14.040 INFO:tasks.ceph.ceph_manager.cluster2:waiting for all up 2024-09-12T06:28:14.040 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-09-12T06:28:14.326 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:28:14.326 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":15,"fsid":"cccfff5c-d879-49a4-9cf1-b06db11d0b7e","created":"2024-09-12T06:27:54.820385+0000","modified":"2024-09-12T06:28:13.355586+0000","last_up_change":"2024-09-12T06:28:04.331593+0000","last_in_change":"2024-09-12T06:27:55.656795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-12T06:28:04.463680+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-12T06:28:09.975795+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cdacbd5a-efce-4370-b990-b1e75740a4cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":2673152699},{"type":"v1","addr":"172.21.15.152:6817","nonce":2673152699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":2673152699},{"type":"v1","addr":"172.21.15.152:6819","nonce":2673152699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":2673152699},{"type":"v1","addr":"172.21.15.152:6823","nonce":2673152699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":2673152699},{"type":"v1","addr":"172.21.15.152:6821","nonce":2673152699}]},"public_addr":"172.21.15.152:6817/2673152699","cluster_addr":"172.21.15.152:6819/2673152699","heartbeat_back_addr":"172.21.15.152:6823/2673152699","heartbeat_front_addr":"172.21.15.152:6821/2673152699","state":["exists","up"]},{"osd":1,"uuid":"9019ee56-f19d-443f-99a7-7e04ef9915b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":1156225285},{"type":"v1","addr":"172.21.15.152:6809","nonce":1156225285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":1156225285},{"type":"v1","addr":"172.21.15.152:6811","nonce":1156225285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":1156225285},{"type":"v1","addr":"172.21.15.152:6815","nonce":1156225285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":1156225285},{"type":"v1","addr":"172.21.15.152:6813","nonce":1156225285}]},"public_addr":"172.21.15.152:6809/1156225285","cluster_addr":"172.21.15.152:6811/1156225285","heartbeat_back_addr":"172.21.15.152:6815/1156225285","heartbeat_front_addr":"172.21.15.152:6813/1156225285","state":["exists","up"]},{"osd":2,"uuid":"47b01d4d-e415-4491-a522-9513d1569287","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":2164790219},{"type":"v1","addr":"172.21.15.152:6801","nonce":2164790219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":2164790219},{"type":"v1","addr":"172.21.15.152:6803","nonce":2164790219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":2164790219},{"type":"v1","addr":"172.21.15.152:6807","nonce":2164790219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":2164790219},{"type":"v1","addr":"172.21.15.152:6805","nonce":2164790219}]},"public_addr":"172.21.15.152:6801/2164790219","cluster_addr":"172.21.15.152:6803/2164790219","heartbeat_back_addr":"172.21.15.152:6807/2164790219","heartbeat_front_addr":"172.21.15.152:6805/2164790219","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T06:28:01.930599+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T06:28:01.728162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T06:28:01.719714+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-12T06:28:14.339 INFO:tasks.ceph.ceph_manager.cluster2:all up! 2024-09-12T06:28:14.339 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-09-12T06:28:14.621 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:28:14.621 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":15,"fsid":"cccfff5c-d879-49a4-9cf1-b06db11d0b7e","created":"2024-09-12T06:27:54.820385+0000","modified":"2024-09-12T06:28:13.355586+0000","last_up_change":"2024-09-12T06:28:04.331593+0000","last_in_change":"2024-09-12T06:27:55.656795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-12T06:28:04.463680+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-12T06:28:09.975795+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cdacbd5a-efce-4370-b990-b1e75740a4cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":2673152699},{"type":"v1","addr":"172.21.15.152:6817","nonce":2673152699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":2673152699},{"type":"v1","addr":"172.21.15.152:6819","nonce":2673152699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":2673152699},{"type":"v1","addr":"172.21.15.152:6823","nonce":2673152699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":2673152699},{"type":"v1","addr":"172.21.15.152:6821","nonce":2673152699}]},"public_addr":"172.21.15.152:6817/2673152699","cluster_addr":"172.21.15.152:6819/2673152699","heartbeat_back_addr":"172.21.15.152:6823/2673152699","heartbeat_front_addr":"172.21.15.152:6821/2673152699","state":["exists","up"]},{"osd":1,"uuid":"9019ee56-f19d-443f-99a7-7e04ef9915b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":1156225285},{"type":"v1","addr":"172.21.15.152:6809","nonce":1156225285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":1156225285},{"type":"v1","addr":"172.21.15.152:6811","nonce":1156225285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":1156225285},{"type":"v1","addr":"172.21.15.152:6815","nonce":1156225285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":1156225285},{"type":"v1","addr":"172.21.15.152:6813","nonce":1156225285}]},"public_addr":"172.21.15.152:6809/1156225285","cluster_addr":"172.21.15.152:6811/1156225285","heartbeat_back_addr":"172.21.15.152:6815/1156225285","heartbeat_front_addr":"172.21.15.152:6813/1156225285","state":["exists","up"]},{"osd":2,"uuid":"47b01d4d-e415-4491-a522-9513d1569287","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":2164790219},{"type":"v1","addr":"172.21.15.152:6801","nonce":2164790219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":2164790219},{"type":"v1","addr":"172.21.15.152:6803","nonce":2164790219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":2164790219},{"type":"v1","addr":"172.21.15.152:6807","nonce":2164790219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":2164790219},{"type":"v1","addr":"172.21.15.152:6805","nonce":2164790219}]},"public_addr":"172.21.15.152:6801/2164790219","cluster_addr":"172.21.15.152:6803/2164790219","heartbeat_back_addr":"172.21.15.152:6807/2164790219","heartbeat_front_addr":"172.21.15.152:6805/2164790219","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T06:28:01.930599+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T06:28:01.728162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T06:28:01.719714+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-12T06:28:14.634 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 flush_pg_stats 2024-09-12T06:28:14.635 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 flush_pg_stats 2024-09-12T06:28:14.635 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 flush_pg_stats 2024-09-12T06:28:14.775 INFO:teuthology.orchestra.run.smithi152.stdout:34359738374 2024-09-12T06:28:14.775 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-09-12T06:28:14.776 INFO:teuthology.orchestra.run.smithi152.stdout:34359738373 2024-09-12T06:28:14.776 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-09-12T06:28:14.777 INFO:teuthology.orchestra.run.smithi152.stdout:34359738374 2024-09-12T06:28:14.777 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-09-12T06:28:15.111 INFO:teuthology.orchestra.run.smithi152.stdout:34359738373 2024-09-12T06:28:15.124 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738373 for osd.0 2024-09-12T06:28:15.134 INFO:teuthology.orchestra.run.smithi152.stdout:34359738372 2024-09-12T06:28:15.137 INFO:teuthology.orchestra.run.smithi152.stdout:34359738373 2024-09-12T06:28:15.148 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738372 for osd.1 2024-09-12T06:28:15.149 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738373 for osd.2 2024-09-12T06:28:16.125 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-09-12T06:28:16.149 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-09-12T06:28:16.151 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-09-12T06:28:16.444 INFO:teuthology.orchestra.run.smithi152.stdout:34359738375 2024-09-12T06:28:16.457 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738375 for osd.0 2024-09-12T06:28:16.457 DEBUG:teuthology.parallel:result is None 2024-09-12T06:28:16.475 INFO:teuthology.orchestra.run.smithi152.stdout:34359738375 2024-09-12T06:28:16.481 INFO:teuthology.orchestra.run.smithi152.stdout:34359738374 2024-09-12T06:28:16.489 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738375 for osd.2 2024-09-12T06:28:16.489 DEBUG:teuthology.parallel:result is None 2024-09-12T06:28:16.497 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738374 for osd.1 2024-09-12T06:28:16.497 DEBUG:teuthology.parallel:result is None 2024-09-12T06:28:16.497 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2024-09-12T06:28:16.497 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T06:28:16.816 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:28:16.817 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-12T06:28:16.828 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-09-12T06:28:16.248571+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":81592,"kb_used_data":504,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283033928,"statfs":{"total":289910292480,"available":289826742272,"internally_reserved":0,"allocated":516096,"data_stored":1295959,"data_compressed":11786,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.899386"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-12T06:28:13.359110+0000","last_change":"2024-09-12T06:28:13.359110+0000","last_active":"2024-09-12T06:28:13.359110+0000","last_peered":"2024-09-12T06:28:13.359110+0000","last_clean":"2024-09-12T06:28:13.359110+0000","last_became_active":"2024-09-12T06:28:11.358431+0000","last_became_peered":"2024-09-12T06:28:11.358431+0000","last_unstale":"2024-09-12T06:28:13.359110+0000","last_undegraded":"2024-09-12T06:28:13.359110+0000","last_fullsized":"2024-09-12T06:28:13.359110+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_clean_scrub_stamp":"2024-09-12T06:28:10.346475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:44.092579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089666099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-12T06:28:13.359336+0000","last_change":"2024-09-12T06:28:13.359336+0000","last_active":"2024-09-12T06:28:13.359336+0000","last_peered":"2024-09-12T06:28:13.359336+0000","last_clean":"2024-09-12T06:28:13.359336+0000","last_became_active":"2024-09-12T06:28:11.358327+0000","last_became_peered":"2024-09-12T06:28:11.358327+0000","last_unstale":"2024-09-12T06:28:13.359336+0000","last_undegraded":"2024-09-12T06:28:13.359336+0000","last_fullsized":"2024-09-12T06:28:13.359336+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_clean_scrub_stamp":"2024-09-12T06:28:10.346475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:08:25.715384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001063589,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-12T06:28:13.358717+0000","last_change":"2024-09-12T06:28:13.358717+0000","last_active":"2024-09-12T06:28:13.358717+0000","last_peered":"2024-09-12T06:28:13.358717+0000","last_clean":"2024-09-12T06:28:13.358717+0000","last_became_active":"2024-09-12T06:28:11.358302+0000","last_became_peered":"2024-09-12T06:28:11.358302+0000","last_unstale":"2024-09-12T06:28:13.358717+0000","last_undegraded":"2024-09-12T06:28:13.358717+0000","last_fullsized":"2024-09-12T06:28:13.358717+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_clean_scrub_stamp":"2024-09-12T06:28:10.346475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T17:13:31.101064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055283899999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-12T06:28:13.358819+0000","last_change":"2024-09-12T06:28:13.358819+0000","last_active":"2024-09-12T06:28:13.358819+0000","last_peered":"2024-09-12T06:28:13.358819+0000","last_clean":"2024-09-12T06:28:13.358819+0000","last_became_active":"2024-09-12T06:28:11.357578+0000","last_became_peered":"2024-09-12T06:28:11.357578+0000","last_unstale":"2024-09-12T06:28:13.358819+0000","last_undegraded":"2024-09-12T06:28:13.358819+0000","last_fullsized":"2024-09-12T06:28:13.358819+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_clean_scrub_stamp":"2024-09-12T06:28:10.346475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:54:00.421871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000609568,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"15'2","reported_seq":22,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-12T06:28:13.365482+0000","last_change":"2024-09-12T06:28:13.364918+0000","last_active":"2024-09-12T06:28:13.365482+0000","last_peered":"2024-09-12T06:28:13.365482+0000","last_clean":"2024-09-12T06:28:13.365482+0000","last_became_active":"2024-09-12T06:28:11.357233+0000","last_became_peered":"2024-09-12T06:28:11.357233+0000","last_unstale":"2024-09-12T06:28:13.365482+0000","last_undegraded":"2024-09-12T06:28:13.365482+0000","last_fullsized":"2024-09-12T06:28:13.365482+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_clean_scrub_stamp":"2024-09-12T06:28:10.346475+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:37:00.884436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012297149999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-12T06:28:13.473086+0000","last_change":"2024-09-12T06:28:13.473086+0000","last_active":"2024-09-12T06:28:13.473086+0000","last_peered":"2024-09-12T06:28:13.473086+0000","last_clean":"2024-09-12T06:28:13.473086+0000","last_became_active":"2024-09-12T06:28:11.357446+0000","last_became_peered":"2024-09-12T06:28:11.357446+0000","last_unstale":"2024-09-12T06:28:13.473086+0000","last_undegraded":"2024-09-12T06:28:13.473086+0000","last_fullsized":"2024-09-12T06:28:13.473086+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_clean_scrub_stamp":"2024-09-12T06:28:10.346475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T16:51:04.766222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041499400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-12T06:28:13.473086+0000","last_change":"2024-09-12T06:28:13.473086+0000","last_active":"2024-09-12T06:28:13.473086+0000","last_peered":"2024-09-12T06:28:13.473086+0000","last_clean":"2024-09-12T06:28:13.473086+0000","last_became_active":"2024-09-12T06:28:11.359644+0000","last_became_peered":"2024-09-12T06:28:11.359644+0000","last_unstale":"2024-09-12T06:28:13.473086+0000","last_undegraded":"2024-09-12T06:28:13.473086+0000","last_fullsized":"2024-09-12T06:28:13.473086+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_clean_scrub_stamp":"2024-09-12T06:28:10.346475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T16:48:40.580401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000404719,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"13'1","reported_seq":21,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-12T06:28:13.359050+0000","last_change":"2024-09-12T06:28:13.359050+0000","last_active":"2024-09-12T06:28:13.359050+0000","last_peered":"2024-09-12T06:28:13.359050+0000","last_clean":"2024-09-12T06:28:13.359050+0000","last_became_active":"2024-09-12T06:28:11.358377+0000","last_became_peered":"2024-09-12T06:28:11.358377+0000","last_unstale":"2024-09-12T06:28:13.359050+0000","last_undegraded":"2024-09-12T06:28:13.359050+0000","last_fullsized":"2024-09-12T06:28:13.359050+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_clean_scrub_stamp":"2024-09-12T06:28:10.346475+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:05:39.794241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084816700000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"10'52","reported_seq":88,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-12T06:28:13.358070+0000","last_change":"2024-09-12T06:28:06.474810+0000","last_active":"2024-09-12T06:28:13.358070+0000","last_peered":"2024-09-12T06:28:13.358070+0000","last_clean":"2024-09-12T06:28:13.358070+0000","last_became_active":"2024-09-12T06:28:06.473701+0000","last_became_peered":"2024-09-12T06:28:06.473701+0000","last_unstale":"2024-09-12T06:28:13.358070+0000","last_undegraded":"2024-09-12T06:28:13.358070+0000","last_fullsized":"2024-09-12T06:28:13.358070+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:28:05.332439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:28:05.332439+0000","last_clean_scrub_stamp":"2024-09-12T06:28:05.332439+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T16:09:15.324127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11786,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738375,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":131072,"data_stored":38395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":192512,"data_stored":628782,"data_compressed":5893,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738375,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":192512,"data_stored":628782,"data_compressed":5893,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5893,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5893,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T06:28:16.829 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T06:28:17.111 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:28:17.111 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-12T06:28:17.123 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-09-12T06:28:16.248571+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":81592,"kb_used_data":504,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283033928,"statfs":{"total":289910292480,"available":289826742272,"internally_reserved":0,"allocated":516096,"data_stored":1295959,"data_compressed":11786,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.899386"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-12T06:28:13.359110+0000","last_change":"2024-09-12T06:28:13.359110+0000","last_active":"2024-09-12T06:28:13.359110+0000","last_peered":"2024-09-12T06:28:13.359110+0000","last_clean":"2024-09-12T06:28:13.359110+0000","last_became_active":"2024-09-12T06:28:11.358431+0000","last_became_peered":"2024-09-12T06:28:11.358431+0000","last_unstale":"2024-09-12T06:28:13.359110+0000","last_undegraded":"2024-09-12T06:28:13.359110+0000","last_fullsized":"2024-09-12T06:28:13.359110+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_clean_scrub_stamp":"2024-09-12T06:28:10.346475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:44.092579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089666099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-12T06:28:13.359336+0000","last_change":"2024-09-12T06:28:13.359336+0000","last_active":"2024-09-12T06:28:13.359336+0000","last_peered":"2024-09-12T06:28:13.359336+0000","last_clean":"2024-09-12T06:28:13.359336+0000","last_became_active":"2024-09-12T06:28:11.358327+0000","last_became_peered":"2024-09-12T06:28:11.358327+0000","last_unstale":"2024-09-12T06:28:13.359336+0000","last_undegraded":"2024-09-12T06:28:13.359336+0000","last_fullsized":"2024-09-12T06:28:13.359336+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_clean_scrub_stamp":"2024-09-12T06:28:10.346475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:08:25.715384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001063589,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-12T06:28:13.358717+0000","last_change":"2024-09-12T06:28:13.358717+0000","last_active":"2024-09-12T06:28:13.358717+0000","last_peered":"2024-09-12T06:28:13.358717+0000","last_clean":"2024-09-12T06:28:13.358717+0000","last_became_active":"2024-09-12T06:28:11.358302+0000","last_became_peered":"2024-09-12T06:28:11.358302+0000","last_unstale":"2024-09-12T06:28:13.358717+0000","last_undegraded":"2024-09-12T06:28:13.358717+0000","last_fullsized":"2024-09-12T06:28:13.358717+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_clean_scrub_stamp":"2024-09-12T06:28:10.346475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T17:13:31.101064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055283899999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-12T06:28:13.358819+0000","last_change":"2024-09-12T06:28:13.358819+0000","last_active":"2024-09-12T06:28:13.358819+0000","last_peered":"2024-09-12T06:28:13.358819+0000","last_clean":"2024-09-12T06:28:13.358819+0000","last_became_active":"2024-09-12T06:28:11.357578+0000","last_became_peered":"2024-09-12T06:28:11.357578+0000","last_unstale":"2024-09-12T06:28:13.358819+0000","last_undegraded":"2024-09-12T06:28:13.358819+0000","last_fullsized":"2024-09-12T06:28:13.358819+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_clean_scrub_stamp":"2024-09-12T06:28:10.346475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:54:00.421871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000609568,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"15'2","reported_seq":22,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-12T06:28:13.365482+0000","last_change":"2024-09-12T06:28:13.364918+0000","last_active":"2024-09-12T06:28:13.365482+0000","last_peered":"2024-09-12T06:28:13.365482+0000","last_clean":"2024-09-12T06:28:13.365482+0000","last_became_active":"2024-09-12T06:28:11.357233+0000","last_became_peered":"2024-09-12T06:28:11.357233+0000","last_unstale":"2024-09-12T06:28:13.365482+0000","last_undegraded":"2024-09-12T06:28:13.365482+0000","last_fullsized":"2024-09-12T06:28:13.365482+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_clean_scrub_stamp":"2024-09-12T06:28:10.346475+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:37:00.884436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012297149999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-12T06:28:13.473086+0000","last_change":"2024-09-12T06:28:13.473086+0000","last_active":"2024-09-12T06:28:13.473086+0000","last_peered":"2024-09-12T06:28:13.473086+0000","last_clean":"2024-09-12T06:28:13.473086+0000","last_became_active":"2024-09-12T06:28:11.357446+0000","last_became_peered":"2024-09-12T06:28:11.357446+0000","last_unstale":"2024-09-12T06:28:13.473086+0000","last_undegraded":"2024-09-12T06:28:13.473086+0000","last_fullsized":"2024-09-12T06:28:13.473086+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_clean_scrub_stamp":"2024-09-12T06:28:10.346475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T16:51:04.766222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041499400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-12T06:28:13.473086+0000","last_change":"2024-09-12T06:28:13.473086+0000","last_active":"2024-09-12T06:28:13.473086+0000","last_peered":"2024-09-12T06:28:13.473086+0000","last_clean":"2024-09-12T06:28:13.473086+0000","last_became_active":"2024-09-12T06:28:11.359644+0000","last_became_peered":"2024-09-12T06:28:11.359644+0000","last_unstale":"2024-09-12T06:28:13.473086+0000","last_undegraded":"2024-09-12T06:28:13.473086+0000","last_fullsized":"2024-09-12T06:28:13.473086+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_clean_scrub_stamp":"2024-09-12T06:28:10.346475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T16:48:40.580401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000404719,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"13'1","reported_seq":21,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-12T06:28:13.359050+0000","last_change":"2024-09-12T06:28:13.359050+0000","last_active":"2024-09-12T06:28:13.359050+0000","last_peered":"2024-09-12T06:28:13.359050+0000","last_clean":"2024-09-12T06:28:13.359050+0000","last_became_active":"2024-09-12T06:28:11.358377+0000","last_became_peered":"2024-09-12T06:28:11.358377+0000","last_unstale":"2024-09-12T06:28:13.359050+0000","last_undegraded":"2024-09-12T06:28:13.359050+0000","last_fullsized":"2024-09-12T06:28:13.359050+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_clean_scrub_stamp":"2024-09-12T06:28:10.346475+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:05:39.794241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084816700000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"10'52","reported_seq":88,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-12T06:28:13.358070+0000","last_change":"2024-09-12T06:28:06.474810+0000","last_active":"2024-09-12T06:28:13.358070+0000","last_peered":"2024-09-12T06:28:13.358070+0000","last_clean":"2024-09-12T06:28:13.358070+0000","last_became_active":"2024-09-12T06:28:06.473701+0000","last_became_peered":"2024-09-12T06:28:06.473701+0000","last_unstale":"2024-09-12T06:28:13.358070+0000","last_undegraded":"2024-09-12T06:28:13.358070+0000","last_fullsized":"2024-09-12T06:28:13.358070+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:28:05.332439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:28:05.332439+0000","last_clean_scrub_stamp":"2024-09-12T06:28:05.332439+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T16:09:15.324127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11786,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738375,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":131072,"data_stored":38395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":192512,"data_stored":628782,"data_compressed":5893,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738375,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":192512,"data_stored":628782,"data_compressed":5893,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5893,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5893,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T06:28:17.124 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2024-09-12T06:28:17.124 INFO:tasks.ceph:Waiting until ceph cluster cluster2 is healthy... 2024-09-12T06:28:17.124 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy 2024-09-12T06:28:17.124 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 health --format=json 2024-09-12T06:28:17.441 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:28:17.465 INFO:teuthology.orchestra.run.smithi152.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-12T06:28:17.466 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy done 2024-09-12T06:28:17.466 INFO:teuthology.run_tasks:Running task exec... 2024-09-12T06:28:17.477 INFO:teuthology.task.exec:Executing custom commands... 2024-09-12T06:28:17.477 INFO:teuthology.task.exec:Running commands on role cluster1.client.mirror.0 host ubuntu@smithi152.front.sepia.ceph.com 2024-09-12T06:28:17.477 DEBUG:teuthology.orchestra.run.smithi152:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.4 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-12T06:28:17.835 INFO:teuthology.orchestra.run.smithi152.stdout:[client.mirror.4] 2024-09-12T06:28:17.835 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQC1ieJmBN1sDBAAhFeKI2lQpheZO0/jobxfBA== 2024-09-12T06:28:17.835 INFO:teuthology.orchestra.run.smithi152.stdout: caps mon = "profile rbd-mirror" 2024-09-12T06:28:17.835 INFO:teuthology.orchestra.run.smithi152.stdout: caps osd = "profile rbd" 2024-09-12T06:28:17.835 INFO:teuthology.orchestra.run.smithi152.stderr:updated caps for client.mirror.4 2024-09-12T06:28:17.849 DEBUG:teuthology.orchestra.run.smithi152:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.5 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-12T06:28:18.211 INFO:teuthology.orchestra.run.smithi152.stdout:[client.mirror.5] 2024-09-12T06:28:18.211 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQC1ieJmFfcFEhAAI7D8pFJ2mx9KrurLZR2yPA== 2024-09-12T06:28:18.211 INFO:teuthology.orchestra.run.smithi152.stdout: caps mon = "profile rbd-mirror" 2024-09-12T06:28:18.211 INFO:teuthology.orchestra.run.smithi152.stdout: caps osd = "profile rbd" 2024-09-12T06:28:18.212 INFO:teuthology.orchestra.run.smithi152.stderr:updated caps for client.mirror.5 2024-09-12T06:28:18.224 DEBUG:teuthology.orchestra.run.smithi152:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.6 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-12T06:28:18.584 INFO:teuthology.orchestra.run.smithi152.stdout:[client.mirror.6] 2024-09-12T06:28:18.584 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQC1ieJm74GYFxAA2tCGUwBh9iCrmU+QZ7k90g== 2024-09-12T06:28:18.584 INFO:teuthology.orchestra.run.smithi152.stdout: caps mon = "profile rbd-mirror" 2024-09-12T06:28:18.584 INFO:teuthology.orchestra.run.smithi152.stdout: caps osd = "profile rbd" 2024-09-12T06:28:18.584 INFO:teuthology.orchestra.run.smithi152.stderr:updated caps for client.mirror.6 2024-09-12T06:28:18.597 DEBUG:teuthology.orchestra.run.smithi152:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.4 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-12T06:28:18.960 INFO:teuthology.orchestra.run.smithi152.stdout:[client.mirror.4] 2024-09-12T06:28:18.961 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQDcieJmEF5TJxAAdSF4dV0ZbPS+PY4h9zVTww== 2024-09-12T06:28:18.961 INFO:teuthology.orchestra.run.smithi152.stdout: caps mon = "profile rbd-mirror" 2024-09-12T06:28:18.961 INFO:teuthology.orchestra.run.smithi152.stdout: caps osd = "profile rbd" 2024-09-12T06:28:18.964 INFO:teuthology.orchestra.run.smithi152.stderr:updated caps for client.mirror.4 2024-09-12T06:28:18.974 DEBUG:teuthology.orchestra.run.smithi152:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.5 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-12T06:28:19.338 INFO:teuthology.orchestra.run.smithi152.stdout:[client.mirror.5] 2024-09-12T06:28:19.338 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQDcieJmx1noLBAAKJKtbgmDvr6YEuTli5zKpw== 2024-09-12T06:28:19.338 INFO:teuthology.orchestra.run.smithi152.stdout: caps mon = "profile rbd-mirror" 2024-09-12T06:28:19.338 INFO:teuthology.orchestra.run.smithi152.stdout: caps osd = "profile rbd" 2024-09-12T06:28:19.339 INFO:teuthology.orchestra.run.smithi152.stderr:updated caps for client.mirror.5 2024-09-12T06:28:19.352 DEBUG:teuthology.orchestra.run.smithi152:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.6 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-12T06:28:19.718 INFO:teuthology.orchestra.run.smithi152.stdout:[client.mirror.6] 2024-09-12T06:28:19.718 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQDcieJmc8SBMhAAy603eaRrC05q4mPHom0PYg== 2024-09-12T06:28:19.718 INFO:teuthology.orchestra.run.smithi152.stdout: caps mon = "profile rbd-mirror" 2024-09-12T06:28:19.718 INFO:teuthology.orchestra.run.smithi152.stdout: caps osd = "profile rbd" 2024-09-12T06:28:19.718 INFO:teuthology.orchestra.run.smithi152.stderr:updated caps for client.mirror.6 2024-09-12T06:28:19.731 INFO:teuthology.run_tasks:Running task exec... 2024-09-12T06:28:19.740 INFO:teuthology.task.exec:Executing custom commands... 2024-09-12T06:28:19.741 INFO:teuthology.task.exec:Running commands on role cluster1.client.mirror.0 host ubuntu@smithi152.front.sepia.ceph.com 2024-09-12T06:28:19.741 DEBUG:teuthology.orchestra.run.smithi152:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2024-09-12T06:28:20.108 INFO:teuthology.orchestra.run.smithi152.stdout:[client.mirror] 2024-09-12T06:28:20.108 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQC0ieJmPwdoLhAAPLpLzaTwMyMvCwncGi2oNg== 2024-09-12T06:28:20.108 INFO:teuthology.orchestra.run.smithi152.stdout: caps mon = "profile rbd-mirror-peer" 2024-09-12T06:28:20.108 INFO:teuthology.orchestra.run.smithi152.stdout: caps osd = "profile rbd" 2024-09-12T06:28:20.108 INFO:teuthology.orchestra.run.smithi152.stderr:updated caps for client.mirror 2024-09-12T06:28:20.122 DEBUG:teuthology.orchestra.run.smithi152:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-12T06:28:20.480 INFO:teuthology.orchestra.run.smithi152.stdout:[client.mirror.0] 2024-09-12T06:28:20.481 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQC0ieJmay+nMRAAf3FVmXMEA6ZvJq4h/jVJXw== 2024-09-12T06:28:20.481 INFO:teuthology.orchestra.run.smithi152.stdout: caps mon = "profile rbd-mirror" 2024-09-12T06:28:20.481 INFO:teuthology.orchestra.run.smithi152.stdout: caps osd = "profile rbd" 2024-09-12T06:28:20.481 INFO:teuthology.orchestra.run.smithi152.stderr:updated caps for client.mirror.0 2024-09-12T06:28:20.494 DEBUG:teuthology.orchestra.run.smithi152:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-12T06:28:20.859 INFO:teuthology.orchestra.run.smithi152.stdout:[client.mirror.1] 2024-09-12T06:28:20.859 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQC0ieJmCzlHNxAABrlTDk3zlswVIAQSeWfESQ== 2024-09-12T06:28:20.859 INFO:teuthology.orchestra.run.smithi152.stdout: caps mon = "profile rbd-mirror" 2024-09-12T06:28:20.859 INFO:teuthology.orchestra.run.smithi152.stdout: caps osd = "profile rbd" 2024-09-12T06:28:20.859 INFO:teuthology.orchestra.run.smithi152.stderr:updated caps for client.mirror.1 2024-09-12T06:28:20.872 DEBUG:teuthology.orchestra.run.smithi152:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-12T06:28:21.235 INFO:teuthology.orchestra.run.smithi152.stdout:[client.mirror.2] 2024-09-12T06:28:21.235 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQC1ieJm/iU/ARAAMLsUXSa4EFda8SRnJ/Xjhw== 2024-09-12T06:28:21.235 INFO:teuthology.orchestra.run.smithi152.stdout: caps mon = "profile rbd-mirror" 2024-09-12T06:28:21.235 INFO:teuthology.orchestra.run.smithi152.stdout: caps osd = "profile rbd" 2024-09-12T06:28:21.235 INFO:teuthology.orchestra.run.smithi152.stderr:updated caps for client.mirror.2 2024-09-12T06:28:21.248 DEBUG:teuthology.orchestra.run.smithi152:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-12T06:28:21.609 INFO:teuthology.orchestra.run.smithi152.stdout:[client.mirror.3] 2024-09-12T06:28:21.609 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQC1ieJmM57MBhAAJSABvLGi2OTN1Mr9zjRXbw== 2024-09-12T06:28:21.609 INFO:teuthology.orchestra.run.smithi152.stdout: caps mon = "profile rbd-mirror" 2024-09-12T06:28:21.609 INFO:teuthology.orchestra.run.smithi152.stdout: caps osd = "profile rbd" 2024-09-12T06:28:21.610 INFO:teuthology.orchestra.run.smithi152.stderr:updated caps for client.mirror.3 2024-09-12T06:28:21.623 DEBUG:teuthology.orchestra.run.smithi152:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2024-09-12T06:28:21.992 INFO:teuthology.orchestra.run.smithi152.stdout:[client.mirror] 2024-09-12T06:28:21.992 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQDcieJmk2g8CxAA/AalP6wEGk/5Cqwj7LaupA== 2024-09-12T06:28:21.992 INFO:teuthology.orchestra.run.smithi152.stdout: caps mon = "profile rbd-mirror-peer" 2024-09-12T06:28:21.992 INFO:teuthology.orchestra.run.smithi152.stdout: caps osd = "profile rbd" 2024-09-12T06:28:21.992 INFO:teuthology.orchestra.run.smithi152.stderr:updated caps for client.mirror 2024-09-12T06:28:22.005 DEBUG:teuthology.orchestra.run.smithi152:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-12T06:28:22.371 INFO:teuthology.orchestra.run.smithi152.stdout:[client.mirror.0] 2024-09-12T06:28:22.371 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQDcieJm13rOEBAAv8xtt/WFjSnnZ995dHCWKA== 2024-09-12T06:28:22.371 INFO:teuthology.orchestra.run.smithi152.stdout: caps mon = "profile rbd-mirror" 2024-09-12T06:28:22.371 INFO:teuthology.orchestra.run.smithi152.stdout: caps osd = "profile rbd" 2024-09-12T06:28:22.371 INFO:teuthology.orchestra.run.smithi152.stderr:updated caps for client.mirror.0 2024-09-12T06:28:22.385 DEBUG:teuthology.orchestra.run.smithi152:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-12T06:28:22.747 INFO:teuthology.orchestra.run.smithi152.stdout:[client.mirror.1] 2024-09-12T06:28:22.747 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQDcieJmf5V9FhAAkzMzgAWfzmIhg/c6Frz0KQ== 2024-09-12T06:28:22.748 INFO:teuthology.orchestra.run.smithi152.stdout: caps mon = "profile rbd-mirror" 2024-09-12T06:28:22.748 INFO:teuthology.orchestra.run.smithi152.stdout: caps osd = "profile rbd" 2024-09-12T06:28:22.748 INFO:teuthology.orchestra.run.smithi152.stderr:updated caps for client.mirror.1 2024-09-12T06:28:22.761 DEBUG:teuthology.orchestra.run.smithi152:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-12T06:28:23.123 INFO:teuthology.orchestra.run.smithi152.stdout:[client.mirror.2] 2024-09-12T06:28:23.123 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQDcieJmNEIjHBAArkTZWYIeWwB23WCp3SW0eg== 2024-09-12T06:28:23.123 INFO:teuthology.orchestra.run.smithi152.stdout: caps mon = "profile rbd-mirror" 2024-09-12T06:28:23.124 INFO:teuthology.orchestra.run.smithi152.stdout: caps osd = "profile rbd" 2024-09-12T06:28:23.124 INFO:teuthology.orchestra.run.smithi152.stderr:updated caps for client.mirror.2 2024-09-12T06:28:23.137 DEBUG:teuthology.orchestra.run.smithi152:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-12T06:28:23.500 INFO:teuthology.orchestra.run.smithi152.stdout:[client.mirror.3] 2024-09-12T06:28:23.500 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQDcieJmyoe9IRAAV3vMCSa6ylnN0VX3S++mrg== 2024-09-12T06:28:23.500 INFO:teuthology.orchestra.run.smithi152.stdout: caps mon = "profile rbd-mirror" 2024-09-12T06:28:23.500 INFO:teuthology.orchestra.run.smithi152.stdout: caps osd = "profile rbd" 2024-09-12T06:28:23.501 INFO:teuthology.orchestra.run.smithi152.stderr:updated caps for client.mirror.3 2024-09-12T06:28:23.514 INFO:teuthology.run_tasks:Running task workunit... 2024-09-12T06:28:23.529 INFO:tasks.workunit:Pulling workunits from ref d61b08865d4d98738f45ff2e4806d7fb6d03eca1 2024-09-12T06:28:23.530 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-12T06:28:23.530 DEBUG:teuthology.orchestra.run.smithi152:> stat -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-09-12T06:28:23.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-12T06:28:23.545 INFO:teuthology.orchestra.run.smithi152.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.cluster1.mirror': No such file or directory 2024-09-12T06:28:23.545 DEBUG:teuthology.orchestra.run.smithi152:> mkdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-09-12T06:28:23.601 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-09-12T06:28:23.601 DEBUG:teuthology.orchestra.run.smithi152:> cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror && mkdir -- client.mirror 2024-09-12T06:28:23.656 INFO:tasks.workunit:timeout=3h 2024-09-12T06:28:23.656 INFO:tasks.workunit:cleanup=True 2024-09-12T06:28:23.657 DEBUG:teuthology.orchestra.run.smithi152:> rm -rf /home/ubuntu/cephtest/clone.cluster1.client.mirror && git clone https://github.com/ceph/ceph-ci /home/ubuntu/cephtest/clone.cluster1.client.mirror && cd /home/ubuntu/cephtest/clone.cluster1.client.mirror && git checkout d61b08865d4d98738f45ff2e4806d7fb6d03eca1 2024-09-12T06:28:23.712 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:Cloning into '/home/ubuntu/cephtest/clone.cluster1.client.mirror'... 2024-09-12T06:29:17.716 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:Note: switching to 'd61b08865d4d98738f45ff2e4806d7fb6d03eca1'. 2024-09-12T06:29:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: 2024-09-12T06:29:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-12T06:29:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-12T06:29:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:state without impacting any branches by switching back to a branch. 2024-09-12T06:29:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: 2024-09-12T06:29:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-12T06:29:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-12T06:29:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: 2024-09-12T06:29:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: git switch -c 2024-09-12T06:29:17.718 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: 2024-09-12T06:29:17.718 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:Or undo this operation with: 2024-09-12T06:29:17.718 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: 2024-09-12T06:29:17.718 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: git switch - 2024-09-12T06:29:17.718 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: 2024-09-12T06:29:17.718 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-12T06:29:17.718 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: 2024-09-12T06:29:17.718 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:HEAD is now at d61b08865d4 Merge branch 'wip-nbalacha-ns-mirroring' of https://github.com/nbalacha/ceph into HEAD 2024-09-12T06:29:17.720 DEBUG:teuthology.orchestra.run.smithi152:> cd -- /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror 2024-09-12T06:29:17.798 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-12T06:29:17.800 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2024-09-12T06:29:17.800 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-12T06:29:18.429 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-12T06:29:18.494 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-12T06:29:18.532 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2024-09-12T06:29:18.534 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2024-09-12T06:29:18.534 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-12T06:29:18.572 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2024-09-12T06:29:18.575 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-09-12T06:29:18.575 DEBUG:teuthology.orchestra.run.smithi152:> dd if=/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror of=/dev/stdout 2024-09-12T06:29:18.631 INFO:tasks.workunit:Running workunits matching rbd/rbd_mirror_stress.sh on cluster1.client.mirror... 2024-09-12T06:29:18.632 INFO:tasks.workunit:Running workunit rbd/rbd_mirror_stress.sh... 2024-09-12T06:29:18.632 DEBUG:teuthology.orchestra.run.smithi152:workunit test rbd/rbd_mirror_stress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d61b08865d4d98738f45ff2e4806d7fb6d03eca1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS='' RBD_MIRROR_MIN_COMPAT_CLIENT=octopus RBD_MIRROR_USE_EXISTING_CLUSTER=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2024-09-12T06:29:18.690 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ IMAGE_COUNT=50 2024-09-12T06:29:18.690 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ export LOCKDEP=0 2024-09-12T06:29:18.690 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ LOCKDEP=0 2024-09-12T06:29:18.690 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2024-09-12T06:29:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ . /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_helpers.sh 2024-09-12T06:29:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ RBD_MIRROR_INSTANCES=2 2024-09-12T06:29:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ CLUSTER1=cluster1 2024-09-12T06:29:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ CLUSTER2=cluster2 2024-09-12T06:29:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ PEER_CLUSTER_SUFFIX= 2024-09-12T06:29:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ POOL=mirror 2024-09-12T06:29:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ PARENT_POOL=mirror_parent 2024-09-12T06:29:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ NS1=ns1 2024-09-12T06:29:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ NS2=ns2 2024-09-12T06:29:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ TEMPDIR= 2024-09-12T06:29:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ CEPH_ID=mirror 2024-09-12T06:29:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ RBD_IMAGE_FEATURES=layering,exclusive-lock,journaling 2024-09-12T06:29:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ MIRROR_USER_ID_PREFIX=mirror. 2024-09-12T06:29:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ RBD_MIRROR_MODE=journal 2024-09-12T06:29:18.694 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ MIRROR_POOL_MODE=pool 2024-09-12T06:29:18.694 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' journal = snapshot ']' 2024-09-12T06:29:18.694 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ export 'CEPH_ARGS=--id mirror' 2024-09-12T06:29:18.694 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ CEPH_ARGS='--id mirror' 2024-09-12T06:29:18.694 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ LAST_MIRROR_INSTANCE=1 2024-09-12T06:29:18.694 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2024-09-12T06:29:18.694 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ readlink -f /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/../../../src 2024-09-12T06:29:18.694 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror/src 2024-09-12T06:29:18.694 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ CEPH_BIN=. 2024-09-12T06:29:18.694 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ CEPH_SRC=. 2024-09-12T06:29:18.694 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' -e CMakeCache.txt ']' 2024-09-12T06:29:18.695 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 -gt 0 ']' 2024-09-12T06:29:18.695 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ setup 2024-09-12T06:29:18.695 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local c 2024-09-12T06:29:18.696 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ trap 'cleanup $?' INT TERM EXIT 2024-09-12T06:29:18.696 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ setup_tempdir 2024-09-12T06:29:18.696 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n '' ']' 2024-09-12T06:29:18.696 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ mktemp -d 2024-09-12T06:29:18.697 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ TEMPDIR=/tmp/tmp.GzjHZGs9Z0 2024-09-12T06:29:18.697 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -z 1 ']' 2024-09-12T06:29:18.697 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ setup_pools cluster1 cluster2 2024-09-12T06:29:18.697 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:29:18.697 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_cluster=cluster2 2024-09-12T06:29:18.698 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local mon_map_file 2024-09-12T06:29:18.698 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local mon_addr 2024-09-12T06:29:18.698 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local admin_key_file 2024-09-12T06:29:18.698 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local uuid 2024-09-12T06:29:18.698 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ CEPH_ARGS= 2024-09-12T06:29:18.698 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --cluster cluster1 osd pool create mirror 64 64 2024-09-12T06:29:19.987 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:pool 'mirror' already exists 2024-09-12T06:29:19.997 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ CEPH_ARGS= 2024-09-12T06:29:19.998 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --cluster cluster1 osd pool create mirror_parent 64 64 2024-09-12T06:29:20.995 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:pool 'mirror_parent' already exists 2024-09-12T06:29:21.006 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ CEPH_ARGS= 2024-09-12T06:29:21.007 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 pool init mirror 2024-09-12T06:29:23.915 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ CEPH_ARGS= 2024-09-12T06:29:23.916 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 pool init mirror_parent 2024-09-12T06:29:26.924 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n '' ']' 2024-09-12T06:29:26.925 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ CEPH_ARGS= 2024-09-12T06:29:26.925 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 mirror pool enable --site-name cluster1 mirror pool 2024-09-12T06:29:26.986 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 mirror pool enable mirror_parent image 2024-09-12T06:29:27.033 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 namespace create mirror/ns1 2024-09-12T06:29:27.075 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 namespace create mirror/ns2 2024-09-12T06:29:27.115 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns1 pool 2024-09-12T06:29:27.166 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns2 image 2024-09-12T06:29:27.212 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -z ']' 2024-09-12T06:29:27.212 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -z ']' 2024-09-12T06:29:27.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ peer_add cluster1 mirror cluster2 2024-09-12T06:29:27.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:29:27.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:29:27.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:29:27.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:29:27.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local client_cluster=cluster2 2024-09-12T06:29:27.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:29:27.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_cluster=cluster2 2024-09-12T06:29:27.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local uuid_var_name 2024-09-12T06:29:27.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n '' ']' 2024-09-12T06:29:27.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local error_code 2024-09-12T06:29:27.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local peer_uuid 2024-09-12T06:29:27.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 16 32 2024-09-12T06:29:27.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set +e 2024-09-12T06:29:27.214 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror cluster2 2024-09-12T06:29:27.254 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ peer_uuid=24fadfa7-c5d1-40ab-b728-1167726be111 2024-09-12T06:29:27.255 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ error_code=0 2024-09-12T06:29:27.255 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set -e 2024-09-12T06:29:27.255 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 -eq 17 ']' 2024-09-12T06:29:27.255 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 0 -eq 0 2024-09-12T06:29:27.255 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n '' ']' 2024-09-12T06:29:27.255 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:29:27.255 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ peer_add cluster1 mirror_parent cluster2 2024-09-12T06:29:27.255 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:29:27.255 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:29:27.255 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror_parent 2024-09-12T06:29:27.255 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:29:27.255 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local client_cluster=cluster2 2024-09-12T06:29:27.256 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:29:27.256 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_cluster=cluster2 2024-09-12T06:29:27.256 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local uuid_var_name 2024-09-12T06:29:27.256 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n '' ']' 2024-09-12T06:29:27.256 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local error_code 2024-09-12T06:29:27.256 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local peer_uuid 2024-09-12T06:29:27.256 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 16 32 2024-09-12T06:29:27.256 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set +e 2024-09-12T06:29:27.256 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror_parent cluster2 2024-09-12T06:29:27.295 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ peer_uuid=c74a71aa-d227-4eda-8ee9-497032a44796 2024-09-12T06:29:27.296 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ error_code=0 2024-09-12T06:29:27.296 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set -e 2024-09-12T06:29:27.296 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 -eq 17 ']' 2024-09-12T06:29:27.296 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 0 -eq 0 2024-09-12T06:29:27.296 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n '' ']' 2024-09-12T06:29:27.296 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:29:27.296 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ setup_pools cluster2 cluster1 2024-09-12T06:29:27.296 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:29:27.296 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_cluster=cluster1 2024-09-12T06:29:27.296 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local mon_map_file 2024-09-12T06:29:27.296 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local mon_addr 2024-09-12T06:29:27.296 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local admin_key_file 2024-09-12T06:29:27.297 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local uuid 2024-09-12T06:29:27.297 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ CEPH_ARGS= 2024-09-12T06:29:27.297 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --cluster cluster2 osd pool create mirror 64 64 2024-09-12T06:29:28.440 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:pool 'mirror' already exists 2024-09-12T06:29:28.451 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ CEPH_ARGS= 2024-09-12T06:29:28.452 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --cluster cluster2 osd pool create mirror_parent 64 64 2024-09-12T06:29:29.460 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:pool 'mirror_parent' already exists 2024-09-12T06:29:29.471 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ CEPH_ARGS= 2024-09-12T06:29:29.472 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 pool init mirror 2024-09-12T06:29:32.360 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ CEPH_ARGS= 2024-09-12T06:29:32.360 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 pool init mirror_parent 2024-09-12T06:29:35.365 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n '' ']' 2024-09-12T06:29:35.365 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ CEPH_ARGS= 2024-09-12T06:29:35.365 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 mirror pool enable --site-name cluster2 mirror pool 2024-09-12T06:29:35.424 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 mirror pool enable mirror_parent image 2024-09-12T06:29:35.467 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 namespace create mirror/ns1 2024-09-12T06:29:35.507 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 namespace create mirror/ns2 2024-09-12T06:29:35.549 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns1 pool 2024-09-12T06:29:35.598 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns2 image 2024-09-12T06:29:35.647 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -z ']' 2024-09-12T06:29:35.647 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -z ']' 2024-09-12T06:29:35.647 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ peer_add cluster2 mirror cluster1 2024-09-12T06:29:35.647 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:29:35.647 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:29:35.647 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:29:35.647 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:29:35.648 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local client_cluster=cluster1 2024-09-12T06:29:35.648 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:29:35.648 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_cluster=cluster1 2024-09-12T06:29:35.648 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local uuid_var_name 2024-09-12T06:29:35.648 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n '' ']' 2024-09-12T06:29:35.648 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local error_code 2024-09-12T06:29:35.648 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local peer_uuid 2024-09-12T06:29:35.648 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 16 32 2024-09-12T06:29:35.648 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set +e 2024-09-12T06:29:35.648 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror cluster1 2024-09-12T06:29:35.687 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ peer_uuid=8956a8e4-f22d-419e-b04b-a3720af6ec55 2024-09-12T06:29:35.687 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ error_code=0 2024-09-12T06:29:35.687 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set -e 2024-09-12T06:29:35.688 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 -eq 17 ']' 2024-09-12T06:29:35.688 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 0 -eq 0 2024-09-12T06:29:35.688 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n '' ']' 2024-09-12T06:29:35.688 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:29:35.688 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ peer_add cluster2 mirror_parent cluster1 2024-09-12T06:29:35.688 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:29:35.688 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:29:35.688 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror_parent 2024-09-12T06:29:35.688 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:29:35.688 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local client_cluster=cluster1 2024-09-12T06:29:35.688 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:29:35.688 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_cluster=cluster1 2024-09-12T06:29:35.689 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local uuid_var_name 2024-09-12T06:29:35.689 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n '' ']' 2024-09-12T06:29:35.689 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local error_code 2024-09-12T06:29:35.689 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local peer_uuid 2024-09-12T06:29:35.689 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 16 32 2024-09-12T06:29:35.689 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set +e 2024-09-12T06:29:35.689 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror_parent cluster1 2024-09-12T06:29:35.726 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ peer_uuid=e3b8a9c4-895f-4255-9fb6-fa0907017bf7 2024-09-12T06:29:35.726 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ error_code=0 2024-09-12T06:29:35.727 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set -e 2024-09-12T06:29:35.727 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 -eq 17 ']' 2024-09-12T06:29:35.727 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 0 -eq 0 2024-09-12T06:29:35.727 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n '' ']' 2024-09-12T06:29:35.727 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:29:35.727 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n octopus ']' 2024-09-12T06:29:35.727 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ CEPH_ARGS= 2024-09-12T06:29:35.727 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --cluster cluster1 osd set-require-min-compat-client octopus 2024-09-12T06:29:37.709 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:set require_min_compat_client to octopus 2024-09-12T06:29:37.723 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ CEPH_ARGS= 2024-09-12T06:29:37.723 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --cluster cluster2 osd set-require-min-compat-client octopus 2024-09-12T06:29:39.412 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:set require_min_compat_client to octopus 2024-09-12T06:29:39.426 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ start_mirrors cluster1 2024-09-12T06:29:39.427 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:29:39.427 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:29:39.533 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:29:39.533 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ start_mirror cluster1:0 2024-09-12T06:29:39.533 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:29:39.533 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:29:39.533 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:29:39.534 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:29:39.534 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:29:39.534 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:29:39.534 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:29:39.534 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:29:39.534 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:29:39.534 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:29:39.534 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:29:39.534 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:29:39.534 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:29:39.534 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:29:39.534 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd-mirror --cluster cluster1 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2024-09-12T06:29:39.571 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:29:39.571 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ start_mirror cluster1:1 2024-09-12T06:29:39.571 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:1 2024-09-12T06:29:39.571 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:29:39.571 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-12T06:29:39.571 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:1 2024-09-12T06:29:39.571 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:29:39.572 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:29:39.572 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:29:39.572 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=1 2024-09-12T06:29:39.572 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-12T06:29:39.572 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:29:39.572 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:29:39.572 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=1 2024-09-12T06:29:39.572 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:29:39.572 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:29:39.572 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd-mirror --cluster cluster1 --id mirror.1 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2024-09-12T06:29:39.606 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ start_mirrors cluster2 2024-09-12T06:29:39.606 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:29:39.606 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:29:39.607 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:29:39.607 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ start_mirror cluster2:0 2024-09-12T06:29:39.607 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2:0 2024-09-12T06:29:39.607 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:29:39.608 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-09-12T06:29:39.608 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster2:0 2024-09-12T06:29:39.608 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:29:39.608 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:29:39.608 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster2 2024-09-12T06:29:39.608 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:29:39.608 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster2:0 ']' 2024-09-12T06:29:39.608 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster2 2024-09-12T06:29:39.608 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster2 2024-09-12T06:29:39.608 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:29:39.608 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:29:39.608 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:29:39.608 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd-mirror --cluster cluster2 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2024-09-12T06:29:39.645 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:29:39.645 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ start_mirror cluster2:1 2024-09-12T06:29:39.645 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2:1 2024-09-12T06:29:39.645 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:29:39.645 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-09-12T06:29:39.645 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster2:1 2024-09-12T06:29:39.646 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:29:39.646 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:29:39.646 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster2 2024-09-12T06:29:39.646 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=1 2024-09-12T06:29:39.646 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 1 = cluster2:1 ']' 2024-09-12T06:29:39.646 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster2 2024-09-12T06:29:39.646 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster2 2024-09-12T06:29:39.646 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=1 2024-09-12T06:29:39.646 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:29:39.646 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:29:39.646 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd-mirror --cluster cluster2 --id mirror.1 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2024-09-12T06:29:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ testlog 'TEST: add image and test replay after client crashes' 2024-09-12T06:29:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ tee -a /tmp/tmp.GzjHZGs9Z0/rbd-mirror.test.log 2024-09-12T06:29:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ date '+%F %T' 2024-09-12T06:29:39.684 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo 2024-09-12 06:29:39 TEST: add image and test replay after client crashes 2024-09-12T06:29:39.684 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:2024-09-12 06:29:39 TEST: add image and test replay after client crashes 2024-09-12T06:29:39.685 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=test 2024-09-12T06:29:39.685 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image_and_enable_mirror cluster2 mirror test journal 512M 2024-09-12T06:29:39.685 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:29:39.685 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:29:39.685 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:29:39.685 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:29:39.685 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:29:39.685 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:29:39.685 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local mode=journal 2024-09-12T06:29:39.685 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n journal ']' 2024-09-12T06:29:39.685 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:29:39.685 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image cluster2 mirror test 512M 2024-09-12T06:29:39.685 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:29:39.686 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:29:39.686 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:29:39.686 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:29:39.686 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:29:39.686 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:29:39.686 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local size=128 2024-09-12T06:29:39.686 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n 512M ']' 2024-09-12T06:29:39.686 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ size=512M 2024-09-12T06:29:39.686 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:29:39.686 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/test 2024-09-12T06:29:39.763 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T06:29:39.763 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:29:39.763 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' pool = image ']' 2024-09-12T06:29:39.763 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ case "${pool}" in 2024-09-12T06:29:39.763 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 1 2024-09-12T06:29:39.763 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-12T06:29:39.763 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:29:39.763 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:29:39.763 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:29:39.763 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-12T06:29:39.764 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:29:39.764 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:29:39.764 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:29:39.764 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=started 2024-09-12T06:29:39.764 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:29:39.764 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:29:39.764 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:29:40.764 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-12T06:29:40.764 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:29:40.765 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:29:40.765 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:29:40.765 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=started 2024-09-12T06:29:40.765 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local status_result 2024-09-12T06:29:40.765 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=stopped 2024-09-12T06:29:40.765 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-12T06:29:40.765 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_instance=cluster1 2024-09-12T06:29:40.765 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -i state 2024-09-12T06:29:40.765 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:29:40.765 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:29:40.765 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance=cluster1 2024-09-12T06:29:40.766 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local loop_instance 2024-09-12T06:29:40.766 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:29:40.766 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sleep 0 2024-09-12T06:29:40.769 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:29:40.769 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ seq 0 1 2024-09-12T06:29:40.769 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:29:40.769 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-12T06:29:40.769 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:0 2024-09-12T06:29:40.769 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:29:40.769 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:29:40.770 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:29:40.770 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:29:40.770 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:29:40.770 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:29:40.770 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:29:40.770 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:29:40.770 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:29:40.770 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:29:40.770 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:29:40.770 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:29:40.770 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:29:40.770 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:29:40.770 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:0 2024-09-12T06:29:40.770 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:29:40.770 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:29:40.771 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:29:40.771 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:0 2024-09-12T06:29:40.771 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:29:40.771 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:29:40.771 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:29:40.771 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:29:40.771 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T06:29:40.771 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:29:40.771 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:29:40.771 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=0 2024-09-12T06:29:40.771 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=0 2024-09-12T06:29:40.771 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:29:40.790 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:29:40.790 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:29:40.790 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:29:40.790 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-12T06:29:40.867 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-12T06:29:40.867 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T06:29:40.867 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:admin_socket: invalid command 2024-09-12T06:29:40.870 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:29:40.870 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-12T06:29:40.870 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:1 2024-09-12T06:29:40.870 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:29:40.871 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:29:40.871 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T06:29:40.871 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:1 2024-09-12T06:29:40.871 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:29:40.871 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:29:40.871 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:29:40.871 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:29:40.871 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T06:29:40.871 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:29:40.871 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:29:40.871 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=1 2024-09-12T06:29:40.871 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:29:40.871 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T06:29:40.871 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:1 2024-09-12T06:29:40.871 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:29:40.872 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:29:40.872 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T06:29:40.872 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:1 2024-09-12T06:29:40.872 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:29:40.872 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:29:40.872 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:29:40.872 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:29:40.872 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T06:29:40.872 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:29:40.872 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:29:40.872 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=1 2024-09-12T06:29:40.872 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=1 2024-09-12T06:29:40.872 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T06:29:40.889 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:29:40.889 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:29:40.889 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:29:40.889 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-12T06:29:40.961 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-12T06:29:40.961 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T06:29:40.961 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:admin_socket: invalid command 2024-09-12T06:29:40.962 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:29:40.962 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sleep 1 2024-09-12T06:29:41.963 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:29:41.964 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ seq 0 1 2024-09-12T06:29:41.967 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:29:41.967 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-12T06:29:41.967 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:0 2024-09-12T06:29:41.967 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:29:41.967 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:29:41.967 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:29:41.967 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:29:41.967 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:29:41.967 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:29:41.967 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:29:41.967 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:29:41.968 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:29:41.968 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:29:41.968 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:29:41.968 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:29:41.968 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:29:41.968 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:29:41.968 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:0 2024-09-12T06:29:41.968 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:29:41.968 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:29:41.968 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:29:41.968 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:0 2024-09-12T06:29:41.968 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:29:41.968 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:29:41.968 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:29:41.969 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:29:41.969 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T06:29:41.969 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:29:41.969 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:29:41.969 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=0 2024-09-12T06:29:41.969 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=0 2024-09-12T06:29:41.969 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:29:41.986 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:29:41.986 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:29:41.986 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:29:41.986 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-12T06:29:42.059 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-12T06:29:42.059 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T06:29:42.060 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:admin_socket: invalid command 2024-09-12T06:29:42.063 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:29:42.063 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-12T06:29:42.063 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:1 2024-09-12T06:29:42.063 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:29:42.063 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:29:42.063 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T06:29:42.063 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:1 2024-09-12T06:29:42.063 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:29:42.063 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:29:42.063 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:29:42.064 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:29:42.064 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T06:29:42.064 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:29:42.064 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:29:42.064 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=1 2024-09-12T06:29:42.064 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:29:42.064 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T06:29:42.064 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:1 2024-09-12T06:29:42.064 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:29:42.064 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:29:42.064 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T06:29:42.064 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:1 2024-09-12T06:29:42.064 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:29:42.064 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:29:42.065 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:29:42.065 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:29:42.065 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T06:29:42.065 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:29:42.065 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:29:42.065 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=1 2024-09-12T06:29:42.065 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=1 2024-09-12T06:29:42.065 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T06:29:42.081 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:29:42.081 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:29:42.081 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:29:42.081 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-12T06:29:42.153 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-12T06:29:42.154 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T06:29:42.154 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:admin_socket: invalid command 2024-09-12T06:29:42.154 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:29:42.154 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sleep 2 2024-09-12T06:29:44.156 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:29:44.156 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ seq 0 1 2024-09-12T06:29:44.159 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:29:44.159 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-12T06:29:44.159 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:0 2024-09-12T06:29:44.160 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:29:44.160 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:29:44.160 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:29:44.160 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:29:44.160 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:29:44.160 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:29:44.160 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:29:44.160 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:29:44.160 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:29:44.160 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:29:44.161 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:29:44.161 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:29:44.161 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:29:44.161 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:29:44.161 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:0 2024-09-12T06:29:44.161 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:29:44.161 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:29:44.161 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:29:44.161 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:0 2024-09-12T06:29:44.161 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:29:44.161 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:29:44.161 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:29:44.161 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:29:44.161 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T06:29:44.161 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:29:44.162 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:29:44.162 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=0 2024-09-12T06:29:44.162 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=0 2024-09-12T06:29:44.162 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:29:44.181 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:29:44.181 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:29:44.181 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:29:44.181 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-12T06:29:44.270 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ return 0 2024-09-12T06:29:44.271 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T06:29:44.271 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo ' "state": "Replaying"' 2024-09-12T06:29:44.271 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep -i Replaying 2024-09-12T06:29:44.272 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: "state": "Replaying" 2024-09-12T06:29:44.273 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_state=started 2024-09-12T06:29:44.273 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test started = started 2024-09-12T06:29:44.273 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:29:44.273 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ clean_snap_name= 2024-09-12T06:29:44.274 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 1 10 2024-09-12T06:29:44.274 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 10` 2024-09-12T06:29:44.274 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ stress_write_image cluster2 mirror test 2024-09-12T06:29:44.274 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:29:44.274 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:29:44.275 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:29:44.275 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T06:29:44.275 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local duration=8 2024-09-12T06:29:44.275 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set +e 2024-09-12T06:29:44.275 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-12T06:29:45.973 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:100 IOs, 65639424 bytes 2024-09-12T06:29:46.506 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:200 IOs, 85482496 bytes 2024-09-12T06:29:46.702 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:300 IOs, 86094848 bytes 2024-09-12T06:29:46.889 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:400 IOs, 89059328 bytes 2024-09-12T06:29:47.185 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:500 IOs, 95956992 bytes 2024-09-12T06:29:48.333 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:600 IOs, 131084288 bytes 2024-09-12T06:29:50.019 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:700 IOs, 212054016 bytes 2024-09-12T06:29:51.082 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:800 IOs, 241209344 bytes 2024-09-12T06:29:51.992 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:900 IOs, 273968128 bytes 2024-09-12T06:29:52.285 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ error_code=124 2024-09-12T06:29:52.285 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set -e 2024-09-12T06:29:52.285 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T06:29:52.285 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:29:52.285 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-12T06:29:52.285 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:29:52.285 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:29:52.285 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:29:52.286 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state_pattern=up+replaying 2024-09-12T06:29:52.286 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local description_pattern= 2024-09-12T06:29:52.286 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local service_pattern= 2024-09-12T06:29:52.286 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:29:52.286 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:29:53.287 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-12T06:29:53.287 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:29:53.287 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:29:53.287 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:29:53.287 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state_pattern=up+replaying 2024-09-12T06:29:53.287 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local description_pattern= 2024-09-12T06:29:53.287 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local service_pattern= 2024-09-12T06:29:53.287 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local status 2024-09-12T06:29:53.287 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ CEPH_ARGS= 2024-09-12T06:29:53.287 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-12T06:29:53.339 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ status='test: 2024-09-12T06:29:53.340 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: global_id: 5bf4c9df-a9e9-4aaa-9110-4727ba332309 2024-09-12T06:29:53.340 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: state: up+replaying 2024-09-12T06:29:53.340 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}} 2024-09-12T06:29:53.340 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: service: mirror.0 on smithi152 2024-09-12T06:29:53.340 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: last_update: 2024-09-12 06:29:42 2024-09-12T06:29:53.340 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: peer_sites: 2024-09-12T06:29:53.340 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: name: cluster2 2024-09-12T06:29:53.340 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: state: up+stopped 2024-09-12T06:29:53.340 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: description: local image is primary 2024-09-12T06:29:53.340 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: last_update: 2024-09-12 06:29:49' 2024-09-12T06:29:53.340 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep '^ state: .*up+replaying' 2024-09-12T06:29:53.341 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: state: up+replaying 2024-09-12T06:29:53.341 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep '^ description: .*' 2024-09-12T06:29:53.341 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}} 2024-09-12T06:29:53.342 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n '' ']' 2024-09-12T06:29:53.342 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo up+replaying 2024-09-12T06:29:53.342 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep '^up+' 2024-09-12T06:29:53.342 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:up+replaying 2024-09-12T06:29:53.343 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep 'service: *mirror..* on ' 2024-09-12T06:29:53.344 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: service: mirror.0 on smithi152 2024-09-12T06:29:53.344 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local last_update 2024-09-12T06:29:53.344 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-12T06:29:53.346 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ last_update='2024-09-12 06:29:42' 2024-09-12T06:29:53.346 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-12 06:29:42' 2024-09-12T06:29:53.346 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:29:53.346 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:29:53.346 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:29:53.346 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state_pattern=up+replaying 2024-09-12T06:29:53.346 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'prev_last_update=2024-09-12 06:29:42' 2024-09-12T06:29:53.346 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local status 2024-09-12T06:29:53.346 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ CEPH_ARGS= 2024-09-12T06:29:53.347 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-12T06:29:53.407 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ status='OKOKOK142614264mirror.0smithi15219.3.0-4883-gd61b0886trueOK42684271mirror.1smithi15219.3.0-4883-gd61b0886falseOKtest5bf4c9df-a9e9-4aaa-9110-4727ba332309up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}}42614264mirror.0smithi1522024-09-12 06:29:42cluster24b9b47ae-ca94-4dab-91d1-08bce6a93349up+stoppedlocal image is primary2024-09-12 06:29:49' 2024-09-12T06:29:53.407 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local last_update state 2024-09-12T06:29:53.407 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-12T06:29:53.760 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ last_update='2024-09-12 06:29:42' 2024-09-12T06:29:53.760 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-12T06:29:53.762 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ state=up+replaying 2024-09-12T06:29:53.762 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo up+replaying 2024-09-12T06:29:53.762 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep up+replaying 2024-09-12T06:29:53.763 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:up+replaying 2024-09-12T06:29:53.764 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:29:53.764 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:29:53.764 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ snap_name=snap1 2024-09-12T06:29:53.764 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_snap cluster2 mirror test snap1 2024-09-12T06:29:53.764 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:29:53.764 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:29:53.764 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:29:53.764 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap1 2024-09-12T06:29:53.765 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1 --debug-rbd=20 --debug-journaler=20 2024-09-12T06:30:22.368 INFO:tasks.ceph.cluster2.osd.2.smithi152.stderr:2024-09-12T06:30:22.369+0000 7f4778fb8640 -1 bdev(0x56163c600c00 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 1 2024-09-12T06:30:22.368 INFO:tasks.ceph.cluster2.osd.2.smithi152.stderr:2024-09-12T06:30:22.371+0000 7f4774fb0640 -1 bdev(0x56163c600c00 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 3 2024-09-12T06:30:24.364 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-12T06:30:24.365 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:30:24.365 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:30:24.365 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:30:24.365 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-12T06:30:24.365 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:30:24.365 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:30:24.365 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:30:24.365 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=started 2024-09-12T06:30:24.365 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:30:24.365 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:30:24.365 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:30:25.366 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-12T06:30:25.366 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:30:25.366 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:30:25.366 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:30:25.366 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=started 2024-09-12T06:30:25.366 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local status_result 2024-09-12T06:30:25.367 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=stopped 2024-09-12T06:30:25.367 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-12T06:30:25.367 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_instance=cluster1 2024-09-12T06:30:25.367 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:30:25.367 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:30:25.367 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -i state 2024-09-12T06:30:25.367 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance=cluster1 2024-09-12T06:30:25.367 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local loop_instance 2024-09-12T06:30:25.367 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:30:25.367 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sleep 0 2024-09-12T06:30:25.367 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:30:25.370 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ seq 0 1 2024-09-12T06:30:25.371 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:30:25.371 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-12T06:30:25.371 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:0 2024-09-12T06:30:25.371 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:30:25.371 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:30:25.371 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:30:25.371 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:30:25.371 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:30:25.371 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:30:25.371 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:30:25.371 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:30:25.371 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:30:25.371 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:30:25.371 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:30:25.372 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:30:25.372 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:30:25.372 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:30:25.372 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:0 2024-09-12T06:30:25.372 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:30:25.372 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:30:25.372 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:30:25.372 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:0 2024-09-12T06:30:25.372 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:30:25.372 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:30:25.372 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:30:25.372 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:30:25.372 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T06:30:25.372 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:30:25.372 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:30:25.373 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=0 2024-09-12T06:30:25.373 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=0 2024-09-12T06:30:25.373 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:30:25.389 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:30:25.389 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:30:25.389 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:30:25.389 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-12T06:30:25.469 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ return 0 2024-09-12T06:30:25.470 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T06:30:25.470 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo ' "state": "Replaying"' 2024-09-12T06:30:25.470 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep -i Replaying 2024-09-12T06:30:25.471 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: "state": "Replaying" 2024-09-12T06:30:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_state=started 2024-09-12T06:30:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test started = started 2024-09-12T06:30:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:30:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror test 2024-09-12T06:30:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:30:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:30:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:30:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:30:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:30:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' journal = journal ']' 2024-09-12T06:30:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror test 2024-09-12T06:30:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:30:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:30:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:30:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:30:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:30:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-12T06:30:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-12T06:30:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:30:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:30:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:30:25.673 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror test 2024-09-12T06:30:25.674 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:30:25.674 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:30:25.674 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:30:25.674 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:30:25.674 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n test ']' 2024-09-12T06:30:25.674 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-12T06:30:25.674 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-12T06:30:25.674 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:30:25.674 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:30:25.674 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:30:25.674 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:30:25.674 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:30:25.675 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:30:25.675 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:30:25.675 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:30:25.675 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:30:25.678 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:30:25.678 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-12T06:30:25.678 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:30:25.678 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:30:25.678 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:30:25.678 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:30:25.678 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:30:25.678 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:30:25.678 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:30:25.679 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:30:25.679 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:30:25.679 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:30:25.679 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:30:25.679 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:30:25.679 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:30:25.679 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:30:25.680 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:30:25.680 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:30:25.680 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:30:25.680 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:30:25.680 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:30:25.680 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:30:25.680 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:30:25.680 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:30:25.680 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:30:25.680 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:30:25.680 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:30:25.680 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:30:25.680 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:30:25.681 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:30:25.681 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:30:25.681 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:30:25.695 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:30:25.695 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:30:25.696 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:30:25.696 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-12T06:30:25.781 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:30:25.781 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-12T06:30:25.781 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:30:25.781 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:30:25.782 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:30:25.782 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-12T06:30:25.782 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:30:25.782 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:30:25.782 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:30:25.782 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:30:25.782 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-12T06:30:25.782 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-test.status 2024-09-12T06:30:25.782 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:30:25.783 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:30:25.783 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-12T06:30:25.783 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:30:25.828 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 4 2024-09-12T06:30:25.829 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 4 2024-09-12T06:30:25.829 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:30:25.829 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=3, entry_tid=1], [object_number=16, tag_tid=3, entry_tid=0], [object_number=19, tag_tid=2, entry_tid=17375], [object_number=18, tag_tid=2, entry_tid=17374]]], state=connected] 2024-09-12T06:30:25.829 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=17, tag_tid=3, entry_tid=1], [object_number=16, tag_tid=3, entry_tid=0], [object_number=19, tag_tid=2, entry_tid=17375], [object_number=18, tag_tid=2, entry_tid=17374]]], state=connected] 2024-09-12T06:30:25.831 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:30:25.834 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=17, tag_tid=3, entry_tid=1' 2024-09-12T06:30:25.834 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-12T06:30:25.834 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:30:25.834 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:30:25.834 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:30:25.834 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-12T06:30:25.834 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:30:25.834 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:30:25.834 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:30:25.834 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:30:25.834 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-12T06:30:25.834 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-test.status 2024-09-12T06:30:25.834 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:30:25.835 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:30:25.835 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-12T06:30:25.835 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:30:25.880 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 4 2024-09-12T06:30:25.880 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 4 2024-09-12T06:30:25.880 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:30:25.885 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=3, entry_tid=1], [object_number=16, tag_tid=3, entry_tid=0], [object_number=19, tag_tid=2, entry_tid=17375], [object_number=18, tag_tid=2, entry_tid=17374]]], state=connected] 2024-09-12T06:30:25.885 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=17, tag_tid=3, entry_tid=1], [object_number=16, tag_tid=3, entry_tid=0], [object_number=19, tag_tid=2, entry_tid=17375], [object_number=18, tag_tid=2, entry_tid=17374]]], state=connected] 2024-09-12T06:30:25.891 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:30:25.893 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=17, tag_tid=3, entry_tid=1' 2024-09-12T06:30:25.893 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=17, tag_tid=3, entry_tid=1' -a 'object_number=17, tag_tid=3, entry_tid=1' = 'object_number=17, tag_tid=3, entry_tid=1' 2024-09-12T06:30:25.893 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:30:25.893 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_snap_present cluster1 mirror test snap1 2024-09-12T06:30:25.893 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:30:25.893 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:30:25.894 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:30:25.894 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap1 2024-09-12T06:30:25.894 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:30:25.894 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T06:30:25.894 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:30:26.894 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 info mirror/test@snap1 2024-09-12T06:30:26.945 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:rbd image 'test': 2024-09-12T06:30:26.946 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: size 512 MiB in 128 objects 2024-09-12T06:30:26.946 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: order 22 (4 MiB objects) 2024-09-12T06:30:26.946 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: snapshot_count: 1 2024-09-12T06:30:26.946 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: id: 10a8e43c53b8 2024-09-12T06:30:26.946 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: block_name_prefix: rbd_data.10a8e43c53b8 2024-09-12T06:30:26.946 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: format: 2 2024-09-12T06:30:26.946 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: features: layering, exclusive-lock, journaling 2024-09-12T06:30:26.946 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: op_features: 2024-09-12T06:30:26.946 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: flags: 2024-09-12T06:30:26.946 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: create_timestamp: Thu Sep 12 06:29:41 2024 2024-09-12T06:30:26.946 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: access_timestamp: Thu Sep 12 06:29:41 2024 2024-09-12T06:30:26.946 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: modify_timestamp: Thu Sep 12 06:29:41 2024 2024-09-12T06:30:26.946 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: protected: False 2024-09-12T06:30:26.946 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: journal: 10a8e43c53b8 2024-09-12T06:30:26.947 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring state: enabled 2024-09-12T06:30:26.947 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring mode: journal 2024-09-12T06:30:26.947 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring global id: 5bf4c9df-a9e9-4aaa-9110-4727ba332309 2024-09-12T06:30:26.947 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring primary: false 2024-09-12T06:30:26.950 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:30:26.950 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n '' ']' 2024-09-12T06:30:26.950 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror test snap1 2024-09-12T06:30:26.950 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:30:26.950 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:30:26.950 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap1 2024-09-12T06:30:26.950 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:30:26.950 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export 2024-09-12T06:30:26.951 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:30:26.951 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:30:26.951 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export 2024-09-12T06:30:27.379 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T06:30:27.386 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:30:27.665 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:30:27.667+0000 7fa315eae640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-09-12T06:30:27.665 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:30:27.667+0000 7fa311ea6640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-09-12T06:30:27.944 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T06:30:27.949 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:30:28.159 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:30:28.274 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:30:28.274 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ clean_snap_name=snap1-clean 2024-09-12T06:30:28.274 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_snap cluster2 mirror test snap1-clean 2024-09-12T06:30:28.274 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:30:28.274 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:30:28.274 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:30:28.274 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap1-clean 2024-09-12T06:30:28.275 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1-clean --debug-rbd=20 --debug-journaler=20 2024-09-12T06:30:29.374 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 10` 2024-09-12T06:30:29.375 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ stress_write_image cluster2 mirror test 2024-09-12T06:30:29.375 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:30:29.375 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:30:29.375 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:30:29.375 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T06:30:29.376 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local duration=5 2024-09-12T06:30:29.376 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set +e 2024-09-12T06:30:29.376 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-12T06:30:30.138 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:100 IOs, 30620672 bytes 2024-09-12T06:30:31.750 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:200 IOs, 81585152 bytes 2024-09-12T06:30:33.045 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:300 IOs, 128262144 bytes 2024-09-12T06:30:33.175 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:400 IOs, 128862208 bytes 2024-09-12T06:30:34.384 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ error_code=124 2024-09-12T06:30:34.384 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set -e 2024-09-12T06:30:34.385 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T06:30:34.385 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:30:34.385 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-12T06:30:34.385 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:30:34.385 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:30:34.385 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:30:34.385 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state_pattern=up+replaying 2024-09-12T06:30:34.385 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local description_pattern= 2024-09-12T06:30:34.385 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local service_pattern= 2024-09-12T06:30:34.385 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:30:34.385 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:30:35.386 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-12T06:30:35.386 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:30:35.386 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:30:35.386 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:30:35.386 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state_pattern=up+replaying 2024-09-12T06:30:35.386 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local description_pattern= 2024-09-12T06:30:35.386 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local service_pattern= 2024-09-12T06:30:35.386 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local status 2024-09-12T06:30:35.387 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ CEPH_ARGS= 2024-09-12T06:30:35.387 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-12T06:30:35.438 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ status='test: 2024-09-12T06:30:35.439 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: global_id: 5bf4c9df-a9e9-4aaa-9110-4727ba332309 2024-09-12T06:30:35.439 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: state: up+replaying 2024-09-12T06:30:35.439 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: description: replaying, {"bytes_per_second":5918812.9,"entries_behind_primary":3485,"entries_per_second":379.1,"non_primary_position":{"entry_tid":6271,"object_number":7,"tag_tid":2},"primary_position":{"entry_tid":9756,"object_number":8,"tag_tid":2},"seconds_until_synced":9} 2024-09-12T06:30:35.439 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: service: mirror.0 on smithi152 2024-09-12T06:30:35.439 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: last_update: 2024-09-12 06:30:09 2024-09-12T06:30:35.439 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: peer_sites: 2024-09-12T06:30:35.439 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: name: cluster2 2024-09-12T06:30:35.439 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: state: up+stopped 2024-09-12T06:30:35.439 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: description: local image is primary 2024-09-12T06:30:35.439 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: last_update: 2024-09-12 06:30:34' 2024-09-12T06:30:35.440 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep '^ state: .*up+replaying' 2024-09-12T06:30:35.440 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: state: up+replaying 2024-09-12T06:30:35.440 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep '^ description: .*' 2024-09-12T06:30:35.441 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: description: replaying, {"bytes_per_second":5918812.9,"entries_behind_primary":3485,"entries_per_second":379.1,"non_primary_position":{"entry_tid":6271,"object_number":7,"tag_tid":2},"primary_position":{"entry_tid":9756,"object_number":8,"tag_tid":2},"seconds_until_synced":9} 2024-09-12T06:30:35.441 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n '' ']' 2024-09-12T06:30:35.441 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo up+replaying 2024-09-12T06:30:35.442 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep '^up+' 2024-09-12T06:30:35.442 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:up+replaying 2024-09-12T06:30:35.442 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep 'service: *mirror..* on ' 2024-09-12T06:30:35.443 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: service: mirror.0 on smithi152 2024-09-12T06:30:35.443 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local last_update 2024-09-12T06:30:35.443 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-12T06:30:35.445 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ last_update='2024-09-12 06:30:09' 2024-09-12T06:30:35.445 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-12 06:30:09' 2024-09-12T06:30:35.445 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:30:35.445 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:30:35.445 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:30:35.446 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state_pattern=up+replaying 2024-09-12T06:30:35.446 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'prev_last_update=2024-09-12 06:30:09' 2024-09-12T06:30:35.446 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local status 2024-09-12T06:30:35.446 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ CEPH_ARGS= 2024-09-12T06:30:35.446 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-12T06:30:35.505 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ status='OKOKOK142614264mirror.0smithi15219.3.0-4883-gd61b0886trueOK42684271mirror.1smithi15219.3.0-4883-gd61b0886falseOKtest5bf4c9df-a9e9-4aaa-9110-4727ba332309up+replayingreplaying, {"bytes_per_second":5918812.9,"entries_behind_primary":3485,"entries_per_second":379.1,"non_primary_position":{"entry_tid":6271,"object_number":7,"tag_tid":2},"primary_position":{"entry_tid":9756,"object_number":8,"tag_tid":2},"seconds_until_synced":9}42614264mirror.0smithi1522024-09-12 06:30:09cluster24b9b47ae-ca94-4dab-91d1-08bce6a93349up+stoppedlocal image is primary2024-09-12 06:30:34' 2024-09-12T06:30:35.505 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local last_update state 2024-09-12T06:30:35.505 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-12T06:30:35.507 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ last_update='2024-09-12 06:30:09' 2024-09-12T06:30:35.507 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-12T06:30:35.508 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ state=up+replaying 2024-09-12T06:30:35.509 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo up+replaying 2024-09-12T06:30:35.509 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep up+replaying 2024-09-12T06:30:35.509 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:up+replaying 2024-09-12T06:30:35.510 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:30:35.510 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:30:35.511 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ snap_name=snap2 2024-09-12T06:30:35.511 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_snap cluster2 mirror test snap2 2024-09-12T06:30:35.511 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:30:35.511 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:30:35.511 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:30:35.511 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap2 2024-09-12T06:30:35.511 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2 --debug-rbd=20 --debug-journaler=20 2024-09-12T06:31:07.435 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-12T06:31:07.435 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:31:07.435 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:31:07.435 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:31:07.435 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-12T06:31:07.436 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:31:07.436 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:31:07.436 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:31:07.436 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=started 2024-09-12T06:31:07.436 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:31:07.436 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:31:07.436 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:31:08.435 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-12T06:31:08.435 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:31:08.436 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:31:08.436 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:31:08.436 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=started 2024-09-12T06:31:08.436 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local status_result 2024-09-12T06:31:08.436 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=stopped 2024-09-12T06:31:08.437 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-12T06:31:08.437 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_instance=cluster1 2024-09-12T06:31:08.437 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:31:08.437 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -i state 2024-09-12T06:31:08.437 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:31:08.437 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance=cluster1 2024-09-12T06:31:08.437 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local loop_instance 2024-09-12T06:31:08.438 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:31:08.438 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sleep 0 2024-09-12T06:31:08.438 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:31:08.438 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ seq 0 1 2024-09-12T06:31:08.440 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:31:08.440 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-12T06:31:08.440 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:0 2024-09-12T06:31:08.440 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:31:08.440 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:31:08.440 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:31:08.440 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:31:08.440 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:31:08.440 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:31:08.440 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:31:08.440 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:31:08.441 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:31:08.441 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:31:08.441 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:31:08.441 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:31:08.441 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:31:08.441 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:31:08.441 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:0 2024-09-12T06:31:08.441 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:31:08.441 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:31:08.441 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:31:08.441 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:0 2024-09-12T06:31:08.443 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:31:08.443 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:31:08.443 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:31:08.443 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:31:08.443 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T06:31:08.443 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:31:08.443 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:31:08.443 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=0 2024-09-12T06:31:08.443 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=0 2024-09-12T06:31:08.443 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:31:08.463 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:31:08.463 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:31:08.463 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:31:08.464 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-12T06:31:08.552 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ return 0 2024-09-12T06:31:08.553 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T06:31:08.553 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo ' "state": "Replaying"' 2024-09-12T06:31:08.553 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep -i Replaying 2024-09-12T06:31:08.553 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: "state": "Replaying" 2024-09-12T06:31:08.555 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_state=started 2024-09-12T06:31:08.555 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test started = started 2024-09-12T06:31:08.555 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:31:08.555 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror test 2024-09-12T06:31:08.555 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:31:08.555 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:31:08.556 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:31:08.556 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:31:08.556 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:31:08.556 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' journal = journal ']' 2024-09-12T06:31:08.556 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror test 2024-09-12T06:31:08.556 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:31:08.556 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:31:08.556 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:31:08.556 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:31:08.556 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:31:08.556 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-12T06:31:08.556 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-12T06:31:08.556 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:31:08.556 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:31:08.556 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:31:08.756 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror test 2024-09-12T06:31:08.756 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:31:08.756 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:31:08.756 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:31:08.757 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:31:08.757 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n test ']' 2024-09-12T06:31:08.757 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-12T06:31:08.757 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-12T06:31:08.757 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:31:08.757 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:31:08.757 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:31:08.757 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:31:08.757 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:31:08.757 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:31:08.757 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:31:08.757 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:31:08.757 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:31:08.760 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:31:08.760 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-12T06:31:08.760 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:31:08.760 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:31:08.761 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:31:08.761 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:31:08.761 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:31:08.761 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:31:08.761 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:31:08.761 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:31:08.761 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:31:08.761 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:31:08.761 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:31:08.761 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:31:08.761 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:31:08.761 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:31:08.761 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:31:08.761 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:31:08.762 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:31:08.762 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:31:08.762 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:31:08.762 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:31:08.762 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:31:08.762 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:31:08.762 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:31:08.762 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:31:08.762 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:31:08.762 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:31:08.762 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:31:08.762 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:31:08.762 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:31:08.762 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:31:08.778 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:31:08.778 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:31:08.778 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:31:08.778 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-12T06:31:08.862 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:31:08.862 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-12T06:31:08.862 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:31:08.862 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:31:08.862 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:31:08.862 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-12T06:31:08.862 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:31:08.862 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:31:08.862 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:31:08.862 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:31:08.862 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-12T06:31:08.862 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-test.status 2024-09-12T06:31:08.862 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:31:08.863 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:31:08.863 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-12T06:31:08.863 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:31:08.909 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 7 2024-09-12T06:31:08.909 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 8 2024-09-12T06:31:08.909 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:31:08.909 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=31, tag_tid=5, entry_tid=10631], [object_number=30, tag_tid=5, entry_tid=10630]]], state=connected] 2024-09-12T06:31:08.909 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=29, tag_tid=5, entry_tid=10633], [object_number=28, tag_tid=5, entry_tid=10632], [object_number=31, tag_tid=5, entry_tid=10631], [object_number=30, tag_tid=5, entry_tid=10630]]], state=connected] 2024-09-12T06:31:08.912 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:31:08.914 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=33, tag_tid=6, entry_tid=1' 2024-09-12T06:31:08.914 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-12T06:31:08.914 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:31:08.914 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:31:08.915 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:31:08.915 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-12T06:31:08.915 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:31:08.915 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:31:08.915 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:31:08.915 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:31:08.915 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-12T06:31:08.915 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-test.status 2024-09-12T06:31:08.915 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:31:08.916 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:31:08.916 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-12T06:31:08.916 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:31:08.961 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 7 2024-09-12T06:31:08.961 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 8 2024-09-12T06:31:08.961 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:31:08.961 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=31, tag_tid=5, entry_tid=10631], [object_number=30, tag_tid=5, entry_tid=10630]]], state=connected] 2024-09-12T06:31:08.961 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=29, tag_tid=5, entry_tid=10633], [object_number=28, tag_tid=5, entry_tid=10632], [object_number=31, tag_tid=5, entry_tid=10631], [object_number=30, tag_tid=5, entry_tid=10630]]], state=connected] 2024-09-12T06:31:08.964 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:31:08.966 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=29, tag_tid=5, entry_tid=10633' 2024-09-12T06:31:08.966 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=33, tag_tid=6, entry_tid=1' -a 'object_number=33, tag_tid=6, entry_tid=1' = 'object_number=29, tag_tid=5, entry_tid=10633' 2024-09-12T06:31:08.966 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 'object_number=29, tag_tid=5, entry_tid=10633' '!=' '' 2024-09-12T06:31:08.967 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ break 2024-09-12T06:31:08.967 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 'object_number=29, tag_tid=5, entry_tid=10633' = '' 2024-09-12T06:31:08.967 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ last_mirror_pos='object_number=29, tag_tid=5, entry_tid=10633' 2024-09-12T06:31:08.967 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=33, tag_tid=6, entry_tid=1' 2024-09-12T06:31:08.967 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-12T06:31:08.967 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:31:08.967 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_tag=6 2024-09-12T06:31:08.968 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=29, tag_tid=5, entry_tid=10633' 2024-09-12T06:31:08.968 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-12T06:31:08.968 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:31:08.969 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_tag=5 2024-09-12T06:31:08.969 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=33, tag_tid=6, entry_tid=1' 2024-09-12T06:31:08.970 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-12T06:31:08.970 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:31:08.970 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_entry=1 2024-09-12T06:31:08.971 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=29, tag_tid=5, entry_tid=10633' 2024-09-12T06:31:08.971 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-12T06:31:08.971 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:31:08.972 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_entry=10633 2024-09-12T06:31:08.972 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 6 = 5 -a 1 -le 10633 2024-09-12T06:31:08.973 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:31:08.973 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:31:08.973 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:31:09.174 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror test 2024-09-12T06:31:09.175 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:31:09.175 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:31:09.175 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:31:09.175 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:31:09.175 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n test ']' 2024-09-12T06:31:09.175 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-12T06:31:09.175 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-12T06:31:09.175 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:31:09.175 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:31:09.175 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:31:09.175 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:31:09.175 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:31:09.175 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:31:09.175 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:31:09.176 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:31:09.176 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:31:09.178 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:31:09.178 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-12T06:31:09.178 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:31:09.179 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:31:09.179 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:31:09.179 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:31:09.179 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:31:09.179 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:31:09.179 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:31:09.179 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:31:09.179 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:31:09.179 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:31:09.179 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:31:09.179 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:31:09.179 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:31:09.179 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:31:09.179 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:31:09.179 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:31:09.180 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:31:09.180 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:31:09.180 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:31:09.180 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:31:09.180 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:31:09.180 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:31:09.180 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:31:09.180 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:31:09.180 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:31:09.180 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:31:09.180 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:31:09.180 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:31:09.180 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:31:09.180 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:31:09.196 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:31:09.196 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:31:09.196 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:31:09.196 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-12T06:31:09.280 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:31:09.280 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-12T06:31:09.280 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:31:09.280 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:31:09.280 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:31:09.280 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-12T06:31:09.280 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:31:09.281 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:31:09.281 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:31:09.281 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:31:09.281 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-12T06:31:09.281 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-test.status 2024-09-12T06:31:09.281 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:31:09.281 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:31:09.282 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-12T06:31:09.282 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:31:09.326 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 7 2024-09-12T06:31:09.326 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 8 2024-09-12T06:31:09.326 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:31:09.326 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=31, tag_tid=5, entry_tid=10631], [object_number=30, tag_tid=5, entry_tid=10630]]], state=connected] 2024-09-12T06:31:09.327 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=31, tag_tid=5, entry_tid=10631], [object_number=30, tag_tid=5, entry_tid=10630]]], state=connected] 2024-09-12T06:31:09.329 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:31:09.331 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=33, tag_tid=6, entry_tid=1' 2024-09-12T06:31:09.331 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-12T06:31:09.332 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:31:09.332 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:31:09.332 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:31:09.332 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-12T06:31:09.332 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:31:09.332 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:31:09.332 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:31:09.332 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:31:09.332 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-12T06:31:09.332 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-test.status 2024-09-12T06:31:09.332 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:31:09.333 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:31:09.333 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-12T06:31:09.333 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:31:09.380 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 7 2024-09-12T06:31:09.381 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 8 2024-09-12T06:31:09.381 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:31:09.381 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=31, tag_tid=5, entry_tid=10631], [object_number=30, tag_tid=5, entry_tid=10630]]], state=connected] 2024-09-12T06:31:09.381 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=31, tag_tid=5, entry_tid=10631], [object_number=30, tag_tid=5, entry_tid=10630]]], state=connected] 2024-09-12T06:31:09.381 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:31:09.384 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=33, tag_tid=6, entry_tid=1' 2024-09-12T06:31:09.384 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=33, tag_tid=6, entry_tid=1' -a 'object_number=33, tag_tid=6, entry_tid=1' = 'object_number=33, tag_tid=6, entry_tid=1' 2024-09-12T06:31:09.384 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:31:09.384 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_snap_present cluster1 mirror test snap2 2024-09-12T06:31:09.384 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:31:09.384 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:31:09.384 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:31:09.385 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap2 2024-09-12T06:31:09.385 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:31:09.385 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T06:31:09.385 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:31:10.384 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 info mirror/test@snap2 2024-09-12T06:31:10.436 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:rbd image 'test': 2024-09-12T06:31:10.436 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: size 512 MiB in 128 objects 2024-09-12T06:31:10.436 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: order 22 (4 MiB objects) 2024-09-12T06:31:10.436 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: snapshot_count: 3 2024-09-12T06:31:10.436 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: id: 10a8e43c53b8 2024-09-12T06:31:10.437 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: block_name_prefix: rbd_data.10a8e43c53b8 2024-09-12T06:31:10.437 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: format: 2 2024-09-12T06:31:10.437 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: features: layering, exclusive-lock, journaling 2024-09-12T06:31:10.437 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: op_features: 2024-09-12T06:31:10.437 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: flags: 2024-09-12T06:31:10.437 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: create_timestamp: Thu Sep 12 06:29:41 2024 2024-09-12T06:31:10.437 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: access_timestamp: Thu Sep 12 06:29:41 2024 2024-09-12T06:31:10.437 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: modify_timestamp: Thu Sep 12 06:30:42 2024 2024-09-12T06:31:10.437 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: protected: False 2024-09-12T06:31:10.437 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: journal: 10a8e43c53b8 2024-09-12T06:31:10.437 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring state: enabled 2024-09-12T06:31:10.437 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring mode: journal 2024-09-12T06:31:10.437 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring global id: 5bf4c9df-a9e9-4aaa-9110-4727ba332309 2024-09-12T06:31:10.438 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring primary: false 2024-09-12T06:31:10.440 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:31:10.441 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n snap1-clean ']' 2024-09-12T06:31:10.441 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror test snap1-clean 2024-09-12T06:31:10.441 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:31:10.441 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:31:10.441 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap1-clean 2024-09-12T06:31:10.441 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:31:10.441 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export 2024-09-12T06:31:10.441 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:31:10.441 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:31:10.441 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export 2024-09-12T06:31:10.857 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T06:31:10.865 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:31:11.153 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:31:11.155+0000 7fa316eb0640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-12T06:31:11.154 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:31:11.155+0000 7fa311ea6640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-12T06:31:11.154 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:31:11.156+0000 7fa317eb2640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-12T06:31:11.155 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:31:11.157+0000 7fa315eae640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-12T06:31:11.413 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T06:31:11.420 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:31:11.630 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:31:11.745 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:31:11.745 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror test snap2 2024-09-12T06:31:11.745 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:31:11.745 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:31:11.745 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap2 2024-09-12T06:31:11.745 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:31:11.745 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export 2024-09-12T06:31:11.745 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:31:11.745 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:31:11.746 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export 2024-09-12T06:31:12.184 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T06:31:12.192 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:31:12.813 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T06:31:12.821 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:31:13.030 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:31:13.144 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:31:13.144 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ clean_snap_name=snap2-clean 2024-09-12T06:31:13.144 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_snap cluster2 mirror test snap2-clean 2024-09-12T06:31:13.144 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:31:13.144 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:31:13.144 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:31:13.144 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap2-clean 2024-09-12T06:31:13.144 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2-clean --debug-rbd=20 --debug-journaler=20 2024-09-12T06:31:13.446 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 10` 2024-09-12T06:31:13.446 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ stress_write_image cluster2 mirror test 2024-09-12T06:31:13.446 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:31:13.446 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:31:13.446 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:31:13.446 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T06:31:13.447 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local duration=5 2024-09-12T06:31:13.447 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set +e 2024-09-12T06:31:13.447 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-12T06:31:14.323 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:100 IOs, 19272704 bytes 2024-09-12T06:31:15.585 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:200 IOs, 51686400 bytes 2024-09-12T06:31:16.090 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:300 IOs, 68932608 bytes 2024-09-12T06:31:16.503 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:400 IOs, 85084160 bytes 2024-09-12T06:31:17.446 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:500 IOs, 108039168 bytes 2024-09-12T06:31:18.456 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ error_code=124 2024-09-12T06:31:18.456 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set -e 2024-09-12T06:31:18.456 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T06:31:18.456 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:31:18.456 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-12T06:31:18.456 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:31:18.456 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:31:18.456 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:31:18.456 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state_pattern=up+replaying 2024-09-12T06:31:18.457 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local description_pattern= 2024-09-12T06:31:18.457 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local service_pattern= 2024-09-12T06:31:18.457 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:31:18.457 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:31:19.457 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-12T06:31:19.457 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:31:19.457 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:31:19.457 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:31:19.458 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state_pattern=up+replaying 2024-09-12T06:31:19.458 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local description_pattern= 2024-09-12T06:31:19.458 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local service_pattern= 2024-09-12T06:31:19.458 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local status 2024-09-12T06:31:19.458 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ CEPH_ARGS= 2024-09-12T06:31:19.458 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-12T06:31:19.513 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ status='test: 2024-09-12T06:31:19.513 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: global_id: 5bf4c9df-a9e9-4aaa-9110-4727ba332309 2024-09-12T06:31:19.513 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: state: up+replaying 2024-09-12T06:31:19.513 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: description: replaying, {"bytes_per_second":3988866.77,"entries_behind_primary":0,"entries_per_second":249.1,"non_primary_position":{"entry_tid":1,"object_number":33,"tag_tid":6},"primary_position":{"entry_tid":1,"object_number":33,"tag_tid":6}} 2024-09-12T06:31:19.513 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: service: mirror.0 on smithi152 2024-09-12T06:31:19.513 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: last_update: 2024-09-12 06:31:09 2024-09-12T06:31:19.513 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: peer_sites: 2024-09-12T06:31:19.514 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: name: cluster2 2024-09-12T06:31:19.514 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: state: up+stopped 2024-09-12T06:31:19.514 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: description: local image is primary 2024-09-12T06:31:19.514 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: last_update: 2024-09-12 06:31:15' 2024-09-12T06:31:19.514 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep '^ state: .*up+replaying' 2024-09-12T06:31:19.514 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: state: up+replaying 2024-09-12T06:31:19.515 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep '^ description: .*' 2024-09-12T06:31:19.515 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: description: replaying, {"bytes_per_second":3988866.77,"entries_behind_primary":0,"entries_per_second":249.1,"non_primary_position":{"entry_tid":1,"object_number":33,"tag_tid":6},"primary_position":{"entry_tid":1,"object_number":33,"tag_tid":6}} 2024-09-12T06:31:19.515 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n '' ']' 2024-09-12T06:31:19.515 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo up+replaying 2024-09-12T06:31:19.516 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep '^up+' 2024-09-12T06:31:19.516 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:up+replaying 2024-09-12T06:31:19.516 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep 'service: *mirror..* on ' 2024-09-12T06:31:19.517 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: service: mirror.0 on smithi152 2024-09-12T06:31:19.518 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local last_update 2024-09-12T06:31:19.518 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-12T06:31:19.519 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ last_update='2024-09-12 06:31:09' 2024-09-12T06:31:19.519 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-12 06:31:09' 2024-09-12T06:31:19.519 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:31:19.520 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:31:19.520 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:31:19.520 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state_pattern=up+replaying 2024-09-12T06:31:19.520 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'prev_last_update=2024-09-12 06:31:09' 2024-09-12T06:31:19.520 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local status 2024-09-12T06:31:19.520 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ CEPH_ARGS= 2024-09-12T06:31:19.520 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-12T06:31:19.576 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ status='OKOKOK142614264mirror.0smithi15219.3.0-4883-gd61b0886trueOK42684271mirror.1smithi15219.3.0-4883-gd61b0886falseOKtest5bf4c9df-a9e9-4aaa-9110-4727ba332309up+replayingreplaying, {"bytes_per_second":3988866.77,"entries_behind_primary":0,"entries_per_second":249.1,"non_primary_position":{"entry_tid":1,"object_number":33,"tag_tid":6},"primary_position":{"entry_tid":1,"object_number":33,"tag_tid":6}}42614264mirror.0smithi1522024-09-12 06:31:09cluster24b9b47ae-ca94-4dab-91d1-08bce6a93349up+stoppedlocal image is primary2024-09-12 06:31:15' 2024-09-12T06:31:19.576 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local last_update state 2024-09-12T06:31:19.576 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-12T06:31:19.578 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ last_update='2024-09-12 06:31:09' 2024-09-12T06:31:19.578 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-12T06:31:19.580 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ state=up+replaying 2024-09-12T06:31:19.580 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo up+replaying 2024-09-12T06:31:19.580 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep up+replaying 2024-09-12T06:31:19.581 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:up+replaying 2024-09-12T06:31:19.582 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:31:19.582 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:31:19.582 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ snap_name=snap3 2024-09-12T06:31:19.582 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_snap cluster2 mirror test snap3 2024-09-12T06:31:19.582 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:31:19.582 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:31:19.582 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:31:19.582 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap3 2024-09-12T06:31:19.583 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3 --debug-rbd=20 --debug-journaler=20 2024-09-12T06:31:51.507 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-12T06:31:51.508 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:31:51.508 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:31:51.508 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:31:51.509 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-12T06:31:51.509 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:31:51.509 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:31:51.509 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:31:51.509 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=started 2024-09-12T06:31:51.509 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:31:51.509 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:31:51.509 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:31:52.508 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-12T06:31:52.508 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:31:52.508 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:31:52.508 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:31:52.508 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=started 2024-09-12T06:31:52.509 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local status_result 2024-09-12T06:31:52.509 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=stopped 2024-09-12T06:31:52.510 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-12T06:31:52.510 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_instance=cluster1 2024-09-12T06:31:52.510 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:31:52.510 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -i state 2024-09-12T06:31:52.510 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:31:52.510 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance=cluster1 2024-09-12T06:31:52.510 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local loop_instance 2024-09-12T06:31:52.510 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:31:52.511 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sleep 0 2024-09-12T06:31:52.511 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:31:52.511 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ seq 0 1 2024-09-12T06:31:52.512 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:31:52.512 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-12T06:31:52.513 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:0 2024-09-12T06:31:52.513 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:31:52.513 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:31:52.513 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:31:52.513 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:31:52.513 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:31:52.513 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:31:52.513 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:31:52.513 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:31:52.513 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:31:52.513 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:31:52.513 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:31:52.513 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:31:52.513 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:31:52.514 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:31:52.515 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:0 2024-09-12T06:31:52.515 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:31:52.515 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:31:52.516 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:31:52.516 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:0 2024-09-12T06:31:52.516 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:31:52.516 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:31:52.516 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:31:52.516 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:31:52.516 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T06:31:52.516 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:31:52.516 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:31:52.516 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=0 2024-09-12T06:31:52.516 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=0 2024-09-12T06:31:52.516 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:31:52.535 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:31:52.535 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:31:52.535 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:31:52.535 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-12T06:31:52.622 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ return 0 2024-09-12T06:31:52.623 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T06:31:52.623 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo ' "state": "Replaying"' 2024-09-12T06:31:52.623 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep -i Replaying 2024-09-12T06:31:52.624 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: "state": "Replaying" 2024-09-12T06:31:52.626 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_state=started 2024-09-12T06:31:52.626 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test started = started 2024-09-12T06:31:52.626 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:31:52.626 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror test 2024-09-12T06:31:52.626 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:31:52.626 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:31:52.626 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:31:52.626 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:31:52.626 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:31:52.626 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' journal = journal ']' 2024-09-12T06:31:52.626 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror test 2024-09-12T06:31:52.627 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:31:52.627 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:31:52.627 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:31:52.627 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:31:52.627 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:31:52.627 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-12T06:31:52.627 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-12T06:31:52.627 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:31:52.627 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:31:52.627 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:31:52.827 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror test 2024-09-12T06:31:52.827 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:31:52.827 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:31:52.827 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:31:52.827 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:31:52.827 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n test ']' 2024-09-12T06:31:52.827 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-12T06:31:52.827 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-12T06:31:52.827 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:31:52.828 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:31:52.828 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:31:52.828 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:31:52.828 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:31:52.828 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:31:52.828 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:31:52.828 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:31:52.828 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:31:52.831 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:31:52.831 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-12T06:31:52.831 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:31:52.831 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:31:52.831 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:31:52.831 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:31:52.832 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:31:52.832 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:31:52.832 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:31:52.832 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:31:52.832 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:31:52.832 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:31:52.832 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:31:52.832 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:31:52.832 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:31:52.832 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:31:52.832 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:31:52.832 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:31:52.832 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:31:52.833 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:31:52.833 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:31:52.833 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:31:52.833 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:31:52.833 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:31:52.833 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:31:52.833 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:31:52.833 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:31:52.833 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:31:52.833 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:31:52.833 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:31:52.833 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:31:52.833 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:31:52.850 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:31:52.850 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:31:52.851 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:31:52.851 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-12T06:31:52.935 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:31:52.936 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-12T06:31:52.936 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:31:52.936 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:31:52.936 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:31:52.936 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-12T06:31:52.936 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:31:52.936 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:31:52.936 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:31:52.936 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:31:52.936 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-12T06:31:52.936 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-test.status 2024-09-12T06:31:52.936 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:31:52.937 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:31:52.937 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-12T06:31:52.937 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:31:52.983 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 10 2024-09-12T06:31:52.983 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 10 2024-09-12T06:31:52.983 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:31:52.984 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=41, tag_tid=9, entry_tid=1], [object_number=40, tag_tid=9, entry_tid=0], [object_number=43, tag_tid=8, entry_tid=8939], [object_number=42, tag_tid=8, entry_tid=8938]]], state=connected] 2024-09-12T06:31:52.984 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=40, tag_tid=8, entry_tid=8940], [object_number=43, tag_tid=8, entry_tid=8939], [object_number=42, tag_tid=8, entry_tid=8938], [object_number=41, tag_tid=8, entry_tid=8937]]], state=connected] 2024-09-12T06:31:52.986 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:31:52.987 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=41, tag_tid=9, entry_tid=1' 2024-09-12T06:31:52.989 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-12T06:31:52.989 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:31:52.989 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:31:52.989 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:31:52.989 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-12T06:31:52.989 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:31:52.989 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:31:52.989 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:31:52.989 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:31:52.990 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-12T06:31:52.990 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-test.status 2024-09-12T06:31:52.990 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:31:52.990 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:31:52.990 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-12T06:31:52.990 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:31:53.035 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 10 2024-09-12T06:31:53.035 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 10 2024-09-12T06:31:53.035 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:31:53.035 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=41, tag_tid=9, entry_tid=1], [object_number=40, tag_tid=9, entry_tid=0], [object_number=43, tag_tid=8, entry_tid=8939], [object_number=42, tag_tid=8, entry_tid=8938]]], state=connected] 2024-09-12T06:31:53.035 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=40, tag_tid=8, entry_tid=8940], [object_number=43, tag_tid=8, entry_tid=8939], [object_number=42, tag_tid=8, entry_tid=8938], [object_number=41, tag_tid=8, entry_tid=8937]]], state=connected] 2024-09-12T06:31:53.038 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:31:53.040 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=40, tag_tid=8, entry_tid=8940' 2024-09-12T06:31:53.040 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=41, tag_tid=9, entry_tid=1' -a 'object_number=41, tag_tid=9, entry_tid=1' = 'object_number=40, tag_tid=8, entry_tid=8940' 2024-09-12T06:31:53.040 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 'object_number=40, tag_tid=8, entry_tid=8940' '!=' '' 2024-09-12T06:31:53.040 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ break 2024-09-12T06:31:53.040 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 'object_number=40, tag_tid=8, entry_tid=8940' = '' 2024-09-12T06:31:53.040 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ last_mirror_pos='object_number=40, tag_tid=8, entry_tid=8940' 2024-09-12T06:31:53.041 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=41, tag_tid=9, entry_tid=1' 2024-09-12T06:31:53.041 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-12T06:31:53.041 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:31:53.041 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_tag=9 2024-09-12T06:31:53.042 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=40, tag_tid=8, entry_tid=8940' 2024-09-12T06:31:53.042 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-12T06:31:53.042 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:31:53.042 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_tag=8 2024-09-12T06:31:53.043 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=41, tag_tid=9, entry_tid=1' 2024-09-12T06:31:53.043 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-12T06:31:53.043 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:31:53.044 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_entry=1 2024-09-12T06:31:53.045 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=40, tag_tid=8, entry_tid=8940' 2024-09-12T06:31:53.045 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-12T06:31:53.045 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:31:53.046 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_entry=8940 2024-09-12T06:31:53.046 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 9 = 8 -a 1 -le 8940 2024-09-12T06:31:53.046 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:31:53.046 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:31:53.046 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:31:53.248 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror test 2024-09-12T06:31:53.248 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:31:53.248 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:31:53.248 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:31:53.248 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:31:53.248 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n test ']' 2024-09-12T06:31:53.248 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-12T06:31:53.249 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-12T06:31:53.249 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:31:53.249 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:31:53.249 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:31:53.249 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:31:53.249 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:31:53.249 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:31:53.249 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:31:53.249 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:31:53.249 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:31:53.252 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:31:53.252 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-12T06:31:53.252 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:31:53.253 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:31:53.253 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:31:53.253 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:31:53.253 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:31:53.253 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:31:53.253 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:31:53.253 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:31:53.253 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:31:53.253 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:31:53.253 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:31:53.253 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:31:53.253 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:31:53.254 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:31:53.254 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:31:53.254 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:31:53.254 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:31:53.254 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:31:53.254 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:31:53.254 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:31:53.254 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:31:53.254 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:31:53.254 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:31:53.254 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:31:53.254 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:31:53.254 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:31:53.254 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:31:53.255 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:31:53.255 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:31:53.255 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:31:53.269 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:31:53.269 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:31:53.269 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:31:53.269 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-12T06:31:53.355 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:31:53.355 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-12T06:31:53.356 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:31:53.356 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:31:53.356 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:31:53.356 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-12T06:31:53.356 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:31:53.356 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:31:53.356 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:31:53.356 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:31:53.356 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-12T06:31:53.356 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-test.status 2024-09-12T06:31:53.356 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:31:53.357 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:31:53.357 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-12T06:31:53.357 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:31:53.404 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 10 2024-09-12T06:31:53.404 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 10 2024-09-12T06:31:53.404 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:31:53.405 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=41, tag_tid=9, entry_tid=1], [object_number=40, tag_tid=9, entry_tid=0], [object_number=43, tag_tid=8, entry_tid=8939], [object_number=42, tag_tid=8, entry_tid=8938]]], state=connected] 2024-09-12T06:31:53.405 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=41, tag_tid=9, entry_tid=1], [object_number=40, tag_tid=9, entry_tid=0], [object_number=43, tag_tid=8, entry_tid=8939], [object_number=42, tag_tid=8, entry_tid=8938]]], state=connected] 2024-09-12T06:31:53.407 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:31:53.408 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=41, tag_tid=9, entry_tid=1' 2024-09-12T06:31:53.410 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-12T06:31:53.410 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:31:53.410 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:31:53.410 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:31:53.410 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-12T06:31:53.410 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:31:53.410 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:31:53.410 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:31:53.410 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:31:53.410 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-12T06:31:53.411 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-test.status 2024-09-12T06:31:53.411 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:31:53.411 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:31:53.411 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-12T06:31:53.411 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:31:53.456 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 10 2024-09-12T06:31:53.456 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 10 2024-09-12T06:31:53.456 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:31:53.456 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=41, tag_tid=9, entry_tid=1], [object_number=40, tag_tid=9, entry_tid=0], [object_number=43, tag_tid=8, entry_tid=8939], [object_number=42, tag_tid=8, entry_tid=8938]]], state=connected] 2024-09-12T06:31:53.456 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=41, tag_tid=9, entry_tid=1], [object_number=40, tag_tid=9, entry_tid=0], [object_number=43, tag_tid=8, entry_tid=8939], [object_number=42, tag_tid=8, entry_tid=8938]]], state=connected] 2024-09-12T06:31:53.459 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:31:53.461 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=41, tag_tid=9, entry_tid=1' 2024-09-12T06:31:53.461 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=41, tag_tid=9, entry_tid=1' -a 'object_number=41, tag_tid=9, entry_tid=1' = 'object_number=41, tag_tid=9, entry_tid=1' 2024-09-12T06:31:53.461 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:31:53.462 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_snap_present cluster1 mirror test snap3 2024-09-12T06:31:53.462 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:31:53.462 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:31:53.462 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:31:53.462 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap3 2024-09-12T06:31:53.462 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:31:53.462 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T06:31:53.462 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:31:54.462 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 info mirror/test@snap3 2024-09-12T06:31:54.517 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:rbd image 'test': 2024-09-12T06:31:54.517 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: size 512 MiB in 128 objects 2024-09-12T06:31:54.517 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: order 22 (4 MiB objects) 2024-09-12T06:31:54.517 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: snapshot_count: 5 2024-09-12T06:31:54.517 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: id: 10a8e43c53b8 2024-09-12T06:31:54.517 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: block_name_prefix: rbd_data.10a8e43c53b8 2024-09-12T06:31:54.517 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: format: 2 2024-09-12T06:31:54.517 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: features: layering, exclusive-lock, journaling 2024-09-12T06:31:54.518 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: op_features: 2024-09-12T06:31:54.518 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: flags: 2024-09-12T06:31:54.518 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: create_timestamp: Thu Sep 12 06:29:41 2024 2024-09-12T06:31:54.518 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: access_timestamp: Thu Sep 12 06:31:10 2024 2024-09-12T06:31:54.518 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: modify_timestamp: Thu Sep 12 06:30:42 2024 2024-09-12T06:31:54.518 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: protected: False 2024-09-12T06:31:54.518 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: journal: 10a8e43c53b8 2024-09-12T06:31:54.518 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring state: enabled 2024-09-12T06:31:54.518 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring mode: journal 2024-09-12T06:31:54.518 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring global id: 5bf4c9df-a9e9-4aaa-9110-4727ba332309 2024-09-12T06:31:54.518 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring primary: false 2024-09-12T06:31:54.521 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:31:54.521 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n snap2-clean ']' 2024-09-12T06:31:54.522 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror test snap2-clean 2024-09-12T06:31:54.522 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:31:54.522 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:31:54.522 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap2-clean 2024-09-12T06:31:54.522 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:31:54.522 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export 2024-09-12T06:31:54.522 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:31:54.522 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:31:54.522 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export 2024-09-12T06:31:54.973 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T06:31:54.980 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:31:55.442 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:31:55.444+0000 7fa313eaa640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-12T06:31:55.443 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:31:55.445+0000 7fa316eb0640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-09-12T06:31:55.443 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:31:55.445+0000 7fa317eb2640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-12T06:31:55.445 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:31:55.447+0000 7fa312ea8640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-12T06:31:55.563 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T06:31:55.571 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:31:55.782 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:31:55.896 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:31:55.896 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror test snap3 2024-09-12T06:31:55.896 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:31:55.896 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:31:55.896 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap3 2024-09-12T06:31:55.896 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:31:55.896 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export 2024-09-12T06:31:55.896 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:31:55.896 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:31:55.897 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export 2024-09-12T06:31:56.350 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T06:31:56.360 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:31:56.984 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T06:31:56.990 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:31:57.199 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:31:57.313 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:31:57.313 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ clean_snap_name=snap3-clean 2024-09-12T06:31:57.314 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_snap cluster2 mirror test snap3-clean 2024-09-12T06:31:57.314 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:31:57.314 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:31:57.314 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:31:57.314 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap3-clean 2024-09-12T06:31:57.314 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3-clean --debug-rbd=20 --debug-journaler=20 2024-09-12T06:31:57.505 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 10` 2024-09-12T06:31:57.505 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ stress_write_image cluster2 mirror test 2024-09-12T06:31:57.505 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:31:57.505 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:31:57.505 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:31:57.506 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T06:31:57.506 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local duration=9 2024-09-12T06:31:57.506 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set +e 2024-09-12T06:31:57.506 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-12T06:31:58.543 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:100 IOs, 40375296 bytes 2024-09-12T06:31:59.380 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:200 IOs, 67973120 bytes 2024-09-12T06:31:59.943 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:300 IOs, 91305984 bytes 2024-09-12T06:32:00.644 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:400 IOs, 114215936 bytes 2024-09-12T06:32:01.455 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:500 IOs, 133854208 bytes 2024-09-12T06:32:02.495 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:600 IOs, 158484480 bytes 2024-09-12T06:32:05.783 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:700 IOs, 249313280 bytes 2024-09-12T06:32:06.516 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ error_code=124 2024-09-12T06:32:06.516 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set -e 2024-09-12T06:32:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T06:32:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:32:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-12T06:32:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:32:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:32:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:32:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state_pattern=up+replaying 2024-09-12T06:32:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local description_pattern= 2024-09-12T06:32:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local service_pattern= 2024-09-12T06:32:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:32:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:32:07.518 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-12T06:32:07.518 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:32:07.518 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:32:07.518 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:32:07.518 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state_pattern=up+replaying 2024-09-12T06:32:07.518 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local description_pattern= 2024-09-12T06:32:07.518 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local service_pattern= 2024-09-12T06:32:07.518 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local status 2024-09-12T06:32:07.518 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ CEPH_ARGS= 2024-09-12T06:32:07.518 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-12T06:32:07.576 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ status='test: 2024-09-12T06:32:07.577 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: global_id: 5bf4c9df-a9e9-4aaa-9110-4727ba332309 2024-09-12T06:32:07.577 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: state: up+replaying 2024-09-12T06:32:07.577 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: description: replaying, {"bytes_per_second":3653983.87,"entries_behind_primary":0,"entries_per_second":233.77,"non_primary_position":{"entry_tid":5578,"object_number":38,"tag_tid":8},"primary_position":{"entry_tid":1,"object_number":33,"tag_tid":7}} 2024-09-12T06:32:07.577 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: service: mirror.0 on smithi152 2024-09-12T06:32:07.577 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: last_update: 2024-09-12 06:31:39 2024-09-12T06:32:07.577 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: peer_sites: 2024-09-12T06:32:07.577 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: name: cluster2 2024-09-12T06:32:07.577 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: state: up+stopped 2024-09-12T06:32:07.577 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: description: local image is primary 2024-09-12T06:32:07.577 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: last_update: 2024-09-12 06:32:04' 2024-09-12T06:32:07.577 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep '^ state: .*up+replaying' 2024-09-12T06:32:07.578 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: state: up+replaying 2024-09-12T06:32:07.578 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep '^ description: .*' 2024-09-12T06:32:07.578 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: description: replaying, {"bytes_per_second":3653983.87,"entries_behind_primary":0,"entries_per_second":233.77,"non_primary_position":{"entry_tid":5578,"object_number":38,"tag_tid":8},"primary_position":{"entry_tid":1,"object_number":33,"tag_tid":7}} 2024-09-12T06:32:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n '' ']' 2024-09-12T06:32:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo up+replaying 2024-09-12T06:32:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep '^up+' 2024-09-12T06:32:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:up+replaying 2024-09-12T06:32:07.580 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep 'service: *mirror..* on ' 2024-09-12T06:32:07.580 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: service: mirror.0 on smithi152 2024-09-12T06:32:07.581 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local last_update 2024-09-12T06:32:07.581 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-12T06:32:07.583 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ last_update='2024-09-12 06:31:39' 2024-09-12T06:32:07.583 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-12 06:31:39' 2024-09-12T06:32:07.583 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:32:07.583 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:32:07.583 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:32:07.583 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state_pattern=up+replaying 2024-09-12T06:32:07.583 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'prev_last_update=2024-09-12 06:31:39' 2024-09-12T06:32:07.583 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local status 2024-09-12T06:32:07.583 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ CEPH_ARGS= 2024-09-12T06:32:07.584 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-12T06:32:07.644 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ status='OKOKOK142614264mirror.0smithi15219.3.0-4883-gd61b0886trueOK42684271mirror.1smithi15219.3.0-4883-gd61b0886falseOKtest5bf4c9df-a9e9-4aaa-9110-4727ba332309up+replayingreplaying, {"bytes_per_second":3653983.87,"entries_behind_primary":0,"entries_per_second":233.77,"non_primary_position":{"entry_tid":5578,"object_number":38,"tag_tid":8},"primary_position":{"entry_tid":1,"object_number":33,"tag_tid":7}}42614264mirror.0smithi1522024-09-12 06:31:39cluster24b9b47ae-ca94-4dab-91d1-08bce6a93349up+stoppedlocal image is primary2024-09-12 06:32:04' 2024-09-12T06:32:07.644 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local last_update state 2024-09-12T06:32:07.644 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-12T06:32:07.646 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ last_update='2024-09-12 06:31:39' 2024-09-12T06:32:07.646 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-12T06:32:07.648 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ state=up+replaying 2024-09-12T06:32:07.648 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo up+replaying 2024-09-12T06:32:07.648 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep up+replaying 2024-09-12T06:32:07.649 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:up+replaying 2024-09-12T06:32:07.650 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:32:07.650 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:32:07.650 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ snap_name=snap4 2024-09-12T06:32:07.650 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_snap cluster2 mirror test snap4 2024-09-12T06:32:07.650 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:32:07.650 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:32:07.650 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:32:07.650 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap4 2024-09-12T06:32:07.650 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4 --debug-rbd=20 --debug-journaler=20 2024-09-12T06:32:36.569 INFO:tasks.ceph.cluster2.osd.1.smithi152.stderr:2024-09-12T06:32:36.571+0000 7f33e5055640 -1 bdev(0x55c73b90ec00 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 2 2024-09-12T06:32:36.570 INFO:tasks.ceph.cluster2.osd.1.smithi152.stderr:2024-09-12T06:32:36.572+0000 7f33e785a640 -1 bdev(0x55c73b90ec00 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 3 2024-09-12T06:32:38.578 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-12T06:32:38.579 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:32:38.579 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:32:38.579 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:32:38.579 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-12T06:32:38.579 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:32:38.579 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:32:38.579 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:32:38.580 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=started 2024-09-12T06:32:38.580 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:32:38.580 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:32:38.580 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:32:39.579 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-12T06:32:39.579 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:32:39.580 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:32:39.580 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:32:39.580 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=started 2024-09-12T06:32:39.580 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local status_result 2024-09-12T06:32:39.580 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=stopped 2024-09-12T06:32:39.581 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-12T06:32:39.581 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_instance=cluster1 2024-09-12T06:32:39.581 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:32:39.581 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -i state 2024-09-12T06:32:39.582 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:32:39.582 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance=cluster1 2024-09-12T06:32:39.582 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local loop_instance 2024-09-12T06:32:39.582 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:32:39.582 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sleep 0 2024-09-12T06:32:39.582 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:32:39.582 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ seq 0 1 2024-09-12T06:32:39.584 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:32:39.584 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-12T06:32:39.584 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:0 2024-09-12T06:32:39.585 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:32:39.585 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:32:39.585 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:32:39.585 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:32:39.585 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:32:39.585 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:32:39.585 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:32:39.585 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:32:39.585 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:32:39.585 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:32:39.586 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:32:39.586 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:32:39.586 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:32:39.586 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:32:39.586 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:0 2024-09-12T06:32:39.586 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:32:39.586 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:32:39.586 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:32:39.586 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:0 2024-09-12T06:32:39.586 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:32:39.587 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:32:39.588 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:32:39.588 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:32:39.588 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T06:32:39.588 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:32:39.588 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:32:39.588 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=0 2024-09-12T06:32:39.588 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=0 2024-09-12T06:32:39.588 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:32:39.607 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:32:39.607 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:32:39.607 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:32:39.607 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-12T06:32:39.697 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ return 0 2024-09-12T06:32:39.697 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T06:32:39.698 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo ' "state": "Replaying"' 2024-09-12T06:32:39.698 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep -i Replaying 2024-09-12T06:32:39.698 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: "state": "Replaying" 2024-09-12T06:32:39.700 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_state=started 2024-09-12T06:32:39.700 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test started = started 2024-09-12T06:32:39.700 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:32:39.700 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror test 2024-09-12T06:32:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:32:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:32:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:32:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:32:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:32:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' journal = journal ']' 2024-09-12T06:32:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror test 2024-09-12T06:32:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:32:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:32:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:32:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:32:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:32:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-12T06:32:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-12T06:32:39.702 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:32:39.702 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:32:39.702 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:32:39.901 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror test 2024-09-12T06:32:39.901 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:32:39.901 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:32:39.901 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:32:39.901 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:32:39.901 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n test ']' 2024-09-12T06:32:39.901 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-12T06:32:39.902 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-12T06:32:39.902 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:32:39.902 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:32:39.902 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:32:39.902 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:32:39.902 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:32:39.902 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:32:39.902 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:32:39.902 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:32:39.902 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:32:39.905 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:32:39.905 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-12T06:32:39.905 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:32:39.905 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:32:39.905 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:32:39.905 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:32:39.905 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:32:39.905 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:32:39.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:32:39.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:32:39.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:32:39.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:32:39.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:32:39.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:32:39.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:32:39.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:32:39.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:32:39.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:32:39.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:32:39.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:32:39.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:32:39.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:32:39.907 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:32:39.907 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:32:39.907 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:32:39.907 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:32:39.907 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:32:39.907 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:32:39.907 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:32:39.907 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:32:39.907 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:32:39.907 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:32:39.923 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:32:39.923 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:32:39.923 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:32:39.923 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-12T06:32:40.003 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:32:40.004 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-12T06:32:40.004 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:32:40.004 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:32:40.004 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:32:40.004 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-12T06:32:40.004 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:32:40.004 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:32:40.004 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:32:40.004 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:32:40.004 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-12T06:32:40.004 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-test.status 2024-09-12T06:32:40.005 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:32:40.005 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:32:40.005 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-12T06:32:40.005 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:32:40.052 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 15 2024-09-12T06:32:40.052 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 15 2024-09-12T06:32:40.052 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:32:40.052 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=61, tag_tid=12, entry_tid=1], [object_number=60, tag_tid=12, entry_tid=0], [object_number=63, tag_tid=11, entry_tid=16571], [object_number=62, tag_tid=11, entry_tid=16570]]], state=connected] 2024-09-12T06:32:40.052 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=61, tag_tid=12, entry_tid=1], [object_number=60, tag_tid=12, entry_tid=0], [object_number=63, tag_tid=11, entry_tid=16571], [object_number=62, tag_tid=11, entry_tid=16570]]], state=connected] 2024-09-12T06:32:40.055 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:32:40.057 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=61, tag_tid=12, entry_tid=1' 2024-09-12T06:32:40.057 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-12T06:32:40.058 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:32:40.058 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:32:40.058 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:32:40.058 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-12T06:32:40.058 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:32:40.058 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:32:40.058 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:32:40.058 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:32:40.058 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-12T06:32:40.058 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-test.status 2024-09-12T06:32:40.058 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:32:40.059 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:32:40.059 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-12T06:32:40.059 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:32:40.106 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 15 2024-09-12T06:32:40.106 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 15 2024-09-12T06:32:40.106 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:32:40.106 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=61, tag_tid=12, entry_tid=1], [object_number=60, tag_tid=12, entry_tid=0], [object_number=63, tag_tid=11, entry_tid=16571], [object_number=62, tag_tid=11, entry_tid=16570]]], state=connected] 2024-09-12T06:32:40.106 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=61, tag_tid=12, entry_tid=1], [object_number=60, tag_tid=12, entry_tid=0], [object_number=63, tag_tid=11, entry_tid=16571], [object_number=62, tag_tid=11, entry_tid=16570]]], state=connected] 2024-09-12T06:32:40.108 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:32:40.111 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=61, tag_tid=12, entry_tid=1' 2024-09-12T06:32:40.111 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=61, tag_tid=12, entry_tid=1' -a 'object_number=61, tag_tid=12, entry_tid=1' = 'object_number=61, tag_tid=12, entry_tid=1' 2024-09-12T06:32:40.111 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:32:40.111 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_snap_present cluster1 mirror test snap4 2024-09-12T06:32:40.111 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:32:40.111 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:32:40.111 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:32:40.112 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap4 2024-09-12T06:32:40.112 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:32:40.112 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T06:32:40.112 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:32:41.111 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 info mirror/test@snap4 2024-09-12T06:32:41.164 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:rbd image 'test': 2024-09-12T06:32:41.164 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: size 512 MiB in 128 objects 2024-09-12T06:32:41.164 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: order 22 (4 MiB objects) 2024-09-12T06:32:41.164 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: snapshot_count: 7 2024-09-12T06:32:41.164 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: id: 10a8e43c53b8 2024-09-12T06:32:41.164 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: block_name_prefix: rbd_data.10a8e43c53b8 2024-09-12T06:32:41.165 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: format: 2 2024-09-12T06:32:41.165 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: features: layering, exclusive-lock, journaling 2024-09-12T06:32:41.165 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: op_features: 2024-09-12T06:32:41.165 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: flags: 2024-09-12T06:32:41.165 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: create_timestamp: Thu Sep 12 06:29:41 2024 2024-09-12T06:32:41.165 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: access_timestamp: Thu Sep 12 06:31:10 2024 2024-09-12T06:32:41.165 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: modify_timestamp: Thu Sep 12 06:31:59 2024 2024-09-12T06:32:41.165 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: protected: False 2024-09-12T06:32:41.165 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: journal: 10a8e43c53b8 2024-09-12T06:32:41.165 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring state: enabled 2024-09-12T06:32:41.165 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring mode: journal 2024-09-12T06:32:41.166 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring global id: 5bf4c9df-a9e9-4aaa-9110-4727ba332309 2024-09-12T06:32:41.166 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring primary: false 2024-09-12T06:32:41.169 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:32:41.170 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n snap3-clean ']' 2024-09-12T06:32:41.170 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror test snap3-clean 2024-09-12T06:32:41.170 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:32:41.170 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:32:41.170 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap3-clean 2024-09-12T06:32:41.171 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:32:41.171 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export 2024-09-12T06:32:41.171 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:32:41.171 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:32:41.171 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export 2024-09-12T06:32:41.640 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T06:32:41.650 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:32:42.299 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T06:32:42.307 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:32:42.522 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:32:42.639 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:32:42.639 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror test snap4 2024-09-12T06:32:42.639 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:32:42.639 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:32:42.639 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap4 2024-09-12T06:32:42.640 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:32:42.640 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export 2024-09-12T06:32:42.640 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:32:42.640 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:32:42.640 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export 2024-09-12T06:32:43.109 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T06:32:43.118 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:32:43.431 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:32:43.433+0000 7fa313eaa640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-12T06:32:43.909 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T06:32:43.918 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:32:44.126 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:32:44.241 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:32:44.241 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ clean_snap_name=snap4-clean 2024-09-12T06:32:44.241 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_snap cluster2 mirror test snap4-clean 2024-09-12T06:32:44.241 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:32:44.241 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:32:44.241 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:32:44.241 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap4-clean 2024-09-12T06:32:44.241 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4-clean --debug-rbd=20 --debug-journaler=20 2024-09-12T06:32:44.577 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 10` 2024-09-12T06:32:44.577 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ stress_write_image cluster2 mirror test 2024-09-12T06:32:44.577 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:32:44.578 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:32:44.578 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:32:44.578 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T06:32:44.579 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local duration=5 2024-09-12T06:32:44.579 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set +e 2024-09-12T06:32:44.579 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-12T06:32:47.068 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:100 IOs, 59940864 bytes 2024-09-12T06:32:49.592 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ error_code=124 2024-09-12T06:32:49.592 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set -e 2024-09-12T06:32:49.592 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T06:32:49.593 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:32:49.593 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-12T06:32:49.593 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:32:49.593 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:32:49.593 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:32:49.593 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state_pattern=up+replaying 2024-09-12T06:32:49.593 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local description_pattern= 2024-09-12T06:32:49.593 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local service_pattern= 2024-09-12T06:32:49.593 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:32:49.593 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:32:50.594 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-12T06:32:50.594 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:32:50.594 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:32:50.595 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:32:50.595 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state_pattern=up+replaying 2024-09-12T06:32:50.595 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local description_pattern= 2024-09-12T06:32:50.595 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local service_pattern= 2024-09-12T06:32:50.595 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local status 2024-09-12T06:32:50.595 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ CEPH_ARGS= 2024-09-12T06:32:50.595 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-12T06:32:50.649 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ status='test: 2024-09-12T06:32:50.649 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: global_id: 5bf4c9df-a9e9-4aaa-9110-4727ba332309 2024-09-12T06:32:50.649 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: state: up+replaying 2024-09-12T06:32:50.649 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: description: replaying, {"bytes_per_second":6827401.87,"entries_behind_primary":0,"entries_per_second":429.6,"non_primary_position":{"entry_tid":12610,"object_number":58,"tag_tid":11},"primary_position":{"entry_tid":10065,"object_number":53,"tag_tid":11}} 2024-09-12T06:32:50.650 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: service: mirror.0 on smithi152 2024-09-12T06:32:50.650 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: last_update: 2024-09-12 06:32:39 2024-09-12T06:32:50.650 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: peer_sites: 2024-09-12T06:32:50.650 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: name: cluster2 2024-09-12T06:32:50.650 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: state: up+stopped 2024-09-12T06:32:50.650 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: description: local image is primary 2024-09-12T06:32:50.650 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: last_update: 2024-09-12 06:32:49' 2024-09-12T06:32:50.650 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep '^ state: .*up+replaying' 2024-09-12T06:32:50.650 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: state: up+replaying 2024-09-12T06:32:50.651 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep '^ description: .*' 2024-09-12T06:32:50.651 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: description: replaying, {"bytes_per_second":6827401.87,"entries_behind_primary":0,"entries_per_second":429.6,"non_primary_position":{"entry_tid":12610,"object_number":58,"tag_tid":11},"primary_position":{"entry_tid":10065,"object_number":53,"tag_tid":11}} 2024-09-12T06:32:50.652 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n '' ']' 2024-09-12T06:32:50.652 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo up+replaying 2024-09-12T06:32:50.652 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep '^up+' 2024-09-12T06:32:50.652 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:up+replaying 2024-09-12T06:32:50.652 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep 'service: *mirror..* on ' 2024-09-12T06:32:50.653 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: service: mirror.0 on smithi152 2024-09-12T06:32:50.654 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local last_update 2024-09-12T06:32:50.654 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-12T06:32:50.656 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ last_update='2024-09-12 06:32:39' 2024-09-12T06:32:50.656 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-12 06:32:39' 2024-09-12T06:32:50.656 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:32:50.656 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:32:50.656 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:32:50.656 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state_pattern=up+replaying 2024-09-12T06:32:50.656 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'prev_last_update=2024-09-12 06:32:39' 2024-09-12T06:32:50.656 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local status 2024-09-12T06:32:50.657 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ CEPH_ARGS= 2024-09-12T06:32:50.657 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-12T06:32:50.715 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ status='OKOKOK142614264mirror.0smithi15219.3.0-4883-gd61b0886trueOK42684271mirror.1smithi15219.3.0-4883-gd61b0886falseOKtest5bf4c9df-a9e9-4aaa-9110-4727ba332309up+replayingreplaying, {"bytes_per_second":6827401.87,"entries_behind_primary":0,"entries_per_second":429.6,"non_primary_position":{"entry_tid":12610,"object_number":58,"tag_tid":11},"primary_position":{"entry_tid":10065,"object_number":53,"tag_tid":11}}42614264mirror.0smithi1522024-09-12 06:32:39cluster24b9b47ae-ca94-4dab-91d1-08bce6a93349up+stoppedlocal image is primary2024-09-12 06:32:49' 2024-09-12T06:32:50.715 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local last_update state 2024-09-12T06:32:50.715 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-12T06:32:50.716 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ last_update='2024-09-12 06:32:39' 2024-09-12T06:32:50.717 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-12T06:32:50.718 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ state=up+replaying 2024-09-12T06:32:50.718 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo up+replaying 2024-09-12T06:32:50.719 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep up+replaying 2024-09-12T06:32:50.719 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:up+replaying 2024-09-12T06:32:50.721 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:32:50.721 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:32:50.721 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ snap_name=snap5 2024-09-12T06:32:50.721 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_snap cluster2 mirror test snap5 2024-09-12T06:32:50.721 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:32:50.721 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:32:50.721 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:32:50.722 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap5 2024-09-12T06:32:50.722 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5 --debug-rbd=20 --debug-journaler=20 2024-09-12T06:33:21.625 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-12T06:33:21.626 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:33:21.626 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:33:21.626 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:33:21.626 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-12T06:33:21.626 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:33:21.626 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:33:21.626 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:33:21.626 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=started 2024-09-12T06:33:21.626 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:33:21.626 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:33:21.626 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:33:22.626 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-12T06:33:22.627 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:33:22.627 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:33:22.627 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:33:22.627 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=started 2024-09-12T06:33:22.627 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local status_result 2024-09-12T06:33:22.627 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=stopped 2024-09-12T06:33:22.627 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-12T06:33:22.627 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_instance=cluster1 2024-09-12T06:33:22.628 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -i state 2024-09-12T06:33:22.628 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:33:22.628 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:33:22.628 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance=cluster1 2024-09-12T06:33:22.628 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local loop_instance 2024-09-12T06:33:22.628 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:33:22.628 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sleep 0 2024-09-12T06:33:22.632 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:33:22.632 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ seq 0 1 2024-09-12T06:33:22.632 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:33:22.632 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-12T06:33:22.632 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:0 2024-09-12T06:33:22.632 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:33:22.632 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:33:22.632 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:33:22.633 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:33:22.633 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:33:22.633 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:33:22.633 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:33:22.633 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:33:22.633 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:33:22.633 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:33:22.633 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:33:22.633 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:33:22.633 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:33:22.633 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:33:22.633 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:0 2024-09-12T06:33:22.634 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:33:22.634 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:33:22.634 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:33:22.634 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:0 2024-09-12T06:33:22.634 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:33:22.634 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:33:22.634 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:33:22.634 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:33:22.634 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T06:33:22.634 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:33:22.634 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:33:22.635 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=0 2024-09-12T06:33:22.635 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=0 2024-09-12T06:33:22.635 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:33:22.652 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:33:22.652 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:33:22.652 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:33:22.652 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-12T06:33:22.740 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ return 0 2024-09-12T06:33:22.740 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T06:33:22.740 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo ' "state": "Replaying"' 2024-09-12T06:33:22.741 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep -i Replaying 2024-09-12T06:33:22.741 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: "state": "Replaying" 2024-09-12T06:33:22.743 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_state=started 2024-09-12T06:33:22.743 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test started = started 2024-09-12T06:33:22.744 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:33:22.744 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror test 2024-09-12T06:33:22.744 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:33:22.744 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:33:22.744 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:33:22.744 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:33:22.744 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:33:22.744 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' journal = journal ']' 2024-09-12T06:33:22.744 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror test 2024-09-12T06:33:22.744 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:33:22.744 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:33:22.744 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:33:22.745 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:33:22.745 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:33:22.745 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-12T06:33:22.745 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-12T06:33:22.745 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:33:22.745 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:33:22.745 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:33:22.944 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror test 2024-09-12T06:33:22.945 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:33:22.945 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:33:22.945 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:33:22.945 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:33:22.945 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n test ']' 2024-09-12T06:33:22.945 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-12T06:33:22.945 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-12T06:33:22.945 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:33:22.945 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:33:22.945 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:33:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:33:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:33:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:33:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:33:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:33:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:33:22.949 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:33:22.949 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-12T06:33:22.950 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:33:22.950 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:33:22.950 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:33:22.950 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:33:22.950 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:33:22.950 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:33:22.950 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:33:22.950 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:33:22.950 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:33:22.951 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:33:22.951 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:33:22.951 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:33:22.951 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:33:22.951 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:33:22.951 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:33:22.951 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:33:22.951 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:33:22.951 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:33:22.951 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:33:22.951 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:33:22.951 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:33:22.951 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:33:22.951 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:33:22.951 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:33:22.952 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:33:22.952 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:33:22.952 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:33:22.952 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:33:22.952 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:33:22.952 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:33:22.966 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:33:22.966 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:33:22.966 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:33:22.966 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-12T06:33:23.049 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:33:23.050 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-12T06:33:23.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:33:23.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:33:23.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:33:23.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-12T06:33:23.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:33:23.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:33:23.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:33:23.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:33:23.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-12T06:33:23.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-test.status 2024-09-12T06:33:23.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:33:23.052 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:33:23.052 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-12T06:33:23.052 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:33:23.097 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 17 2024-09-12T06:33:23.097 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 18 2024-09-12T06:33:23.097 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:33:23.097 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=15, entry_tid=1], [object_number=68, tag_tid=15, entry_tid=0], [object_number=71, tag_tid=14, entry_tid=7111], [object_number=70, tag_tid=14, entry_tid=7110]]], state=connected] 2024-09-12T06:33:23.097 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=73, tag_tid=15, entry_tid=1], [object_number=68, tag_tid=15, entry_tid=0], [object_number=71, tag_tid=14, entry_tid=7111], [object_number=70, tag_tid=14, entry_tid=7110]]], state=connected] 2024-09-12T06:33:23.100 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:33:23.101 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=73, tag_tid=15, entry_tid=1' 2024-09-12T06:33:23.103 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-12T06:33:23.103 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:33:23.103 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:33:23.103 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:33:23.103 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-12T06:33:23.103 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:33:23.103 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:33:23.103 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:33:23.103 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:33:23.103 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-12T06:33:23.103 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-test.status 2024-09-12T06:33:23.104 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:33:23.104 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:33:23.104 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-12T06:33:23.104 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:33:23.149 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 17 2024-09-12T06:33:23.149 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 18 2024-09-12T06:33:23.149 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:33:23.150 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=15, entry_tid=1], [object_number=68, tag_tid=15, entry_tid=0], [object_number=71, tag_tid=14, entry_tid=7111], [object_number=70, tag_tid=14, entry_tid=7110]]], state=connected] 2024-09-12T06:33:23.150 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=73, tag_tid=15, entry_tid=1], [object_number=68, tag_tid=15, entry_tid=0], [object_number=71, tag_tid=14, entry_tid=7111], [object_number=70, tag_tid=14, entry_tid=7110]]], state=connected] 2024-09-12T06:33:23.152 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:33:23.155 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=73, tag_tid=15, entry_tid=1' 2024-09-12T06:33:23.155 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=73, tag_tid=15, entry_tid=1' -a 'object_number=73, tag_tid=15, entry_tid=1' = 'object_number=73, tag_tid=15, entry_tid=1' 2024-09-12T06:33:23.155 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:33:23.155 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_snap_present cluster1 mirror test snap5 2024-09-12T06:33:23.155 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:33:23.155 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:33:23.155 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:33:23.155 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap5 2024-09-12T06:33:23.155 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:33:23.155 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T06:33:23.155 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:33:24.155 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 info mirror/test@snap5 2024-09-12T06:33:24.210 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:rbd image 'test': 2024-09-12T06:33:24.210 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: size 512 MiB in 128 objects 2024-09-12T06:33:24.210 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: order 22 (4 MiB objects) 2024-09-12T06:33:24.210 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: snapshot_count: 9 2024-09-12T06:33:24.210 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: id: 10a8e43c53b8 2024-09-12T06:33:24.210 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: block_name_prefix: rbd_data.10a8e43c53b8 2024-09-12T06:33:24.210 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: format: 2 2024-09-12T06:33:24.210 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: features: layering, exclusive-lock, journaling 2024-09-12T06:33:24.210 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: op_features: 2024-09-12T06:33:24.210 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: flags: 2024-09-12T06:33:24.210 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: create_timestamp: Thu Sep 12 06:29:41 2024 2024-09-12T06:33:24.211 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: access_timestamp: Thu Sep 12 06:32:41 2024 2024-09-12T06:33:24.211 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: modify_timestamp: Thu Sep 12 06:33:00 2024 2024-09-12T06:33:24.211 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: protected: False 2024-09-12T06:33:24.211 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: journal: 10a8e43c53b8 2024-09-12T06:33:24.211 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring state: enabled 2024-09-12T06:33:24.211 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring mode: journal 2024-09-12T06:33:24.211 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring global id: 5bf4c9df-a9e9-4aaa-9110-4727ba332309 2024-09-12T06:33:24.211 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring primary: false 2024-09-12T06:33:24.214 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:33:24.214 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n snap4-clean ']' 2024-09-12T06:33:24.214 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror test snap4-clean 2024-09-12T06:33:24.214 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:33:24.214 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:33:24.215 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap4-clean 2024-09-12T06:33:24.215 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:33:24.215 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export 2024-09-12T06:33:24.215 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:33:24.215 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:33:24.215 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export 2024-09-12T06:33:24.701 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T06:33:24.710 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:33:25.004 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:33:25.005+0000 7fa3136a9640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-09-12T06:33:25.005 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:33:25.007+0000 7fa317eb2640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-12T06:33:25.125 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:33:25.127+0000 7fa317eb2640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-12T06:33:25.125 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:33:25.127+0000 7fa315eae640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-12T06:33:25.126 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:33:25.128+0000 7fa316eb0640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-12T06:33:25.452 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T06:33:25.462 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:33:25.672 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:33:25.788 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:33:25.788 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror test snap5 2024-09-12T06:33:25.788 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:33:25.788 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:33:25.789 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap5 2024-09-12T06:33:25.789 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:33:25.789 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export 2024-09-12T06:33:25.789 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:33:25.789 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:33:25.789 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export 2024-09-12T06:33:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T06:33:26.294 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:33:26.648 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:33:26.650+0000 7fa316eb0640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-12T06:33:26.648 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:33:26.650+0000 7fa313eaa640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-12T06:33:26.649 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:33:26.651+0000 7fa3166af640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-12T06:33:26.649 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:33:26.651+0000 7fa315eae640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-12T06:33:27.017 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T06:33:27.025 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:33:27.234 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:33:27.350 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:33:27.350 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ clean_snap_name=snap5-clean 2024-09-12T06:33:27.350 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_snap cluster2 mirror test snap5-clean 2024-09-12T06:33:27.350 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:33:27.350 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:33:27.350 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:33:27.350 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap5-clean 2024-09-12T06:33:27.350 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5-clean --debug-rbd=20 --debug-journaler=20 2024-09-12T06:33:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 10` 2024-09-12T06:33:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ stress_write_image cluster2 mirror test 2024-09-12T06:33:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:33:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:33:27.636 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:33:27.636 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T06:33:27.637 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local duration=5 2024-09-12T06:33:27.637 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set +e 2024-09-12T06:33:27.637 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-12T06:33:29.381 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:100 IOs, 43991040 bytes 2024-09-12T06:33:29.771 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:200 IOs, 83355648 bytes 2024-09-12T06:33:31.332 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:300 IOs, 108733440 bytes 2024-09-12T06:33:31.626 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:400 IOs, 131368960 bytes 2024-09-12T06:33:32.644 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ error_code=124 2024-09-12T06:33:32.645 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set -e 2024-09-12T06:33:32.645 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T06:33:32.645 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:33:32.645 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-12T06:33:32.645 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:33:32.645 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:33:32.645 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:33:32.645 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state_pattern=up+replaying 2024-09-12T06:33:32.645 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local description_pattern= 2024-09-12T06:33:32.645 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local service_pattern= 2024-09-12T06:33:32.645 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:33:32.645 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:33:33.646 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-12T06:33:33.646 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:33:33.646 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:33:33.646 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:33:33.646 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state_pattern=up+replaying 2024-09-12T06:33:33.646 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local description_pattern= 2024-09-12T06:33:33.646 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local service_pattern= 2024-09-12T06:33:33.646 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local status 2024-09-12T06:33:33.646 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ CEPH_ARGS= 2024-09-12T06:33:33.646 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-12T06:33:33.698 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ status='test: 2024-09-12T06:33:33.699 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: global_id: 5bf4c9df-a9e9-4aaa-9110-4727ba332309 2024-09-12T06:33:33.699 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: state: up+replaying 2024-09-12T06:33:33.699 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: description: replaying, {"bytes_per_second":3844660.8,"entries_behind_primary":0,"entries_per_second":238.3,"non_primary_position":{"entry_tid":6809,"object_number":69,"tag_tid":14},"primary_position":{"entry_tid":1,"object_number":61,"tag_tid":13}} 2024-09-12T06:33:33.699 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: service: mirror.0 on smithi152 2024-09-12T06:33:33.699 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: last_update: 2024-09-12 06:33:09 2024-09-12T06:33:33.699 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: peer_sites: 2024-09-12T06:33:33.699 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: name: cluster2 2024-09-12T06:33:33.699 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: state: up+stopped 2024-09-12T06:33:33.699 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: description: local image is primary 2024-09-12T06:33:33.699 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: last_update: 2024-09-12 06:33:29' 2024-09-12T06:33:33.699 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep '^ state: .*up+replaying' 2024-09-12T06:33:33.700 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: state: up+replaying 2024-09-12T06:33:33.700 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep '^ description: .*' 2024-09-12T06:33:33.701 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: description: replaying, {"bytes_per_second":3844660.8,"entries_behind_primary":0,"entries_per_second":238.3,"non_primary_position":{"entry_tid":6809,"object_number":69,"tag_tid":14},"primary_position":{"entry_tid":1,"object_number":61,"tag_tid":13}} 2024-09-12T06:33:33.701 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n '' ']' 2024-09-12T06:33:33.701 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo up+replaying 2024-09-12T06:33:33.701 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep '^up+' 2024-09-12T06:33:33.702 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:up+replaying 2024-09-12T06:33:33.702 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep 'service: *mirror..* on ' 2024-09-12T06:33:33.703 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: service: mirror.0 on smithi152 2024-09-12T06:33:33.703 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local last_update 2024-09-12T06:33:33.703 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-12T06:33:33.705 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ last_update='2024-09-12 06:33:09' 2024-09-12T06:33:33.705 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-12 06:33:09' 2024-09-12T06:33:33.705 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:33:33.705 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:33:33.705 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:33:33.705 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state_pattern=up+replaying 2024-09-12T06:33:33.705 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'prev_last_update=2024-09-12 06:33:09' 2024-09-12T06:33:33.706 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local status 2024-09-12T06:33:33.706 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ CEPH_ARGS= 2024-09-12T06:33:33.706 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-12T06:33:33.766 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ status='OKOKOK142614264mirror.0smithi15219.3.0-4883-gd61b0886trueOK42684271mirror.1smithi15219.3.0-4883-gd61b0886falseOKtest5bf4c9df-a9e9-4aaa-9110-4727ba332309up+replayingreplaying, {"bytes_per_second":3844660.8,"entries_behind_primary":0,"entries_per_second":238.3,"non_primary_position":{"entry_tid":6809,"object_number":69,"tag_tid":14},"primary_position":{"entry_tid":1,"object_number":61,"tag_tid":13}}42614264mirror.0smithi1522024-09-12 06:33:09cluster24b9b47ae-ca94-4dab-91d1-08bce6a93349up+stoppedlocal image is primary2024-09-12 06:33:29' 2024-09-12T06:33:33.766 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local last_update state 2024-09-12T06:33:33.766 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-12T06:33:33.768 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ last_update='2024-09-12 06:33:09' 2024-09-12T06:33:33.768 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-12T06:33:33.770 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ state=up+replaying 2024-09-12T06:33:33.770 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo up+replaying 2024-09-12T06:33:33.770 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep up+replaying 2024-09-12T06:33:33.771 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:up+replaying 2024-09-12T06:33:33.772 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:33:33.772 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:33:33.772 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ snap_name=snap6 2024-09-12T06:33:33.772 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_snap cluster2 mirror test snap6 2024-09-12T06:33:33.772 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:33:33.772 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:33:33.772 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:33:33.772 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap6 2024-09-12T06:33:33.772 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6 --debug-rbd=20 --debug-journaler=20 2024-09-12T06:34:05.698 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-12T06:34:05.698 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:34:05.698 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:34:05.698 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:34:05.698 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-12T06:34:05.699 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:34:05.699 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:34:05.699 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:34:05.699 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=started 2024-09-12T06:34:05.699 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:34:05.699 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:34:05.699 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:34:06.698 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-12T06:34:06.699 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:34:06.699 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:34:06.699 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:34:06.699 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=started 2024-09-12T06:34:06.699 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local status_result 2024-09-12T06:34:06.699 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=stopped 2024-09-12T06:34:06.700 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-12T06:34:06.700 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_instance=cluster1 2024-09-12T06:34:06.700 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:34:06.700 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:34:06.701 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance=cluster1 2024-09-12T06:34:06.701 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local loop_instance 2024-09-12T06:34:06.701 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:34:06.701 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -i state 2024-09-12T06:34:06.701 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sleep 0 2024-09-12T06:34:06.701 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:34:06.701 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ seq 0 1 2024-09-12T06:34:06.703 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:34:06.703 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-12T06:34:06.703 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:0 2024-09-12T06:34:06.703 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:34:06.703 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:34:06.703 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:34:06.703 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:34:06.703 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:34:06.704 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:34:06.704 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:34:06.704 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:34:06.704 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:34:06.704 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:34:06.704 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:34:06.704 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:34:06.704 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:34:06.704 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:34:06.704 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:0 2024-09-12T06:34:06.704 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:34:06.704 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:34:06.704 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:34:06.705 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:0 2024-09-12T06:34:06.705 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:34:06.705 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:34:06.706 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:34:06.706 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:34:06.706 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T06:34:06.706 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:34:06.706 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:34:06.706 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=0 2024-09-12T06:34:06.706 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=0 2024-09-12T06:34:06.706 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:34:06.724 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:34:06.725 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:34:06.725 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:34:06.725 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-12T06:34:06.811 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ return 0 2024-09-12T06:34:06.812 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T06:34:06.812 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo ' "state": "Replaying"' 2024-09-12T06:34:06.812 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep -i Replaying 2024-09-12T06:34:06.813 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: "state": "Replaying" 2024-09-12T06:34:06.814 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_state=started 2024-09-12T06:34:06.815 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test started = started 2024-09-12T06:34:06.815 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:34:06.815 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror test 2024-09-12T06:34:06.815 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:34:06.815 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:34:06.815 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:34:06.815 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:34:06.815 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:34:06.815 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' journal = journal ']' 2024-09-12T06:34:06.816 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror test 2024-09-12T06:34:06.816 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:34:06.816 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:34:06.816 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:34:06.816 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:34:06.816 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:34:06.816 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-12T06:34:06.816 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-12T06:34:06.816 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:34:06.816 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:34:06.816 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:34:07.015 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror test 2024-09-12T06:34:07.016 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:34:07.016 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:34:07.016 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:34:07.016 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:34:07.016 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n test ']' 2024-09-12T06:34:07.016 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-12T06:34:07.016 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-12T06:34:07.016 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:34:07.016 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:34:07.016 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:34:07.017 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:34:07.017 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:34:07.017 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:34:07.017 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:34:07.017 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:34:07.017 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:34:07.019 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:34:07.019 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-12T06:34:07.020 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:34:07.020 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:34:07.020 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:34:07.020 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:34:07.020 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:34:07.020 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:34:07.020 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:34:07.020 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:34:07.020 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:34:07.020 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:34:07.020 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:34:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:34:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:34:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:34:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:34:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:34:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:34:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:34:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:34:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:34:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:34:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:34:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:34:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:34:07.022 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:34:07.022 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:34:07.022 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:34:07.022 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:34:07.022 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:34:07.022 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:34:07.038 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:34:07.038 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:34:07.038 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:34:07.038 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-12T06:34:07.121 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:34:07.122 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-12T06:34:07.122 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:34:07.122 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:34:07.122 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:34:07.122 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-12T06:34:07.122 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:34:07.122 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:34:07.122 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:34:07.122 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:34:07.122 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-12T06:34:07.122 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-test.status 2024-09-12T06:34:07.122 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:34:07.123 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:34:07.123 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-12T06:34:07.123 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:34:07.170 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 20 2024-09-12T06:34:07.170 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 20 2024-09-12T06:34:07.170 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:34:07.170 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=81, tag_tid=18, entry_tid=1], [object_number=80, tag_tid=18, entry_tid=0], [object_number=83, tag_tid=17, entry_tid=9015], [object_number=82, tag_tid=17, entry_tid=9014]]], state=connected] 2024-09-12T06:34:07.170 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=81, tag_tid=18, entry_tid=1], [object_number=80, tag_tid=18, entry_tid=0], [object_number=83, tag_tid=17, entry_tid=9015], [object_number=82, tag_tid=17, entry_tid=9014]]], state=connected] 2024-09-12T06:34:07.173 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:34:07.175 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=81, tag_tid=18, entry_tid=1' 2024-09-12T06:34:07.175 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-12T06:34:07.175 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:34:07.175 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:34:07.176 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:34:07.176 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-12T06:34:07.176 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:34:07.176 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:34:07.176 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:34:07.176 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:34:07.176 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-12T06:34:07.176 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-test.status 2024-09-12T06:34:07.176 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:34:07.177 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:34:07.177 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-12T06:34:07.177 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:34:07.222 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 20 2024-09-12T06:34:07.223 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 20 2024-09-12T06:34:07.223 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:34:07.223 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=81, tag_tid=18, entry_tid=1], [object_number=80, tag_tid=18, entry_tid=0], [object_number=83, tag_tid=17, entry_tid=9015], [object_number=82, tag_tid=17, entry_tid=9014]]], state=connected] 2024-09-12T06:34:07.223 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=81, tag_tid=18, entry_tid=1], [object_number=80, tag_tid=18, entry_tid=0], [object_number=83, tag_tid=17, entry_tid=9015], [object_number=82, tag_tid=17, entry_tid=9014]]], state=connected] 2024-09-12T06:34:07.225 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:34:07.227 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=81, tag_tid=18, entry_tid=1' 2024-09-12T06:34:07.228 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=81, tag_tid=18, entry_tid=1' -a 'object_number=81, tag_tid=18, entry_tid=1' = 'object_number=81, tag_tid=18, entry_tid=1' 2024-09-12T06:34:07.228 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:34:07.228 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_snap_present cluster1 mirror test snap6 2024-09-12T06:34:07.228 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:34:07.228 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:34:07.228 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:34:07.228 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap6 2024-09-12T06:34:07.228 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:34:07.229 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T06:34:07.229 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:34:08.228 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 info mirror/test@snap6 2024-09-12T06:34:08.284 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:rbd image 'test': 2024-09-12T06:34:08.284 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: size 512 MiB in 128 objects 2024-09-12T06:34:08.284 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: order 22 (4 MiB objects) 2024-09-12T06:34:08.284 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: snapshot_count: 11 2024-09-12T06:34:08.284 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: id: 10a8e43c53b8 2024-09-12T06:34:08.284 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: block_name_prefix: rbd_data.10a8e43c53b8 2024-09-12T06:34:08.284 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: format: 2 2024-09-12T06:34:08.284 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: features: layering, exclusive-lock, journaling 2024-09-12T06:34:08.284 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: op_features: 2024-09-12T06:34:08.284 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: flags: 2024-09-12T06:34:08.285 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: create_timestamp: Thu Sep 12 06:29:41 2024 2024-09-12T06:34:08.285 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: access_timestamp: Thu Sep 12 06:32:41 2024 2024-09-12T06:34:08.285 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: modify_timestamp: Thu Sep 12 06:33:00 2024 2024-09-12T06:34:08.285 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: protected: False 2024-09-12T06:34:08.285 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: journal: 10a8e43c53b8 2024-09-12T06:34:08.285 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring state: enabled 2024-09-12T06:34:08.285 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring mode: journal 2024-09-12T06:34:08.285 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring global id: 5bf4c9df-a9e9-4aaa-9110-4727ba332309 2024-09-12T06:34:08.285 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring primary: false 2024-09-12T06:34:08.288 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:34:08.288 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n snap5-clean ']' 2024-09-12T06:34:08.288 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror test snap5-clean 2024-09-12T06:34:08.288 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:34:08.288 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:34:08.288 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap5-clean 2024-09-12T06:34:08.288 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:34:08.288 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export 2024-09-12T06:34:08.288 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:34:08.289 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:34:08.289 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export 2024-09-12T06:34:08.792 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T06:34:08.801 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:34:09.066 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:34:09.067+0000 7fa311ea6640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-12T06:34:09.066 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:34:09.068+0000 7fa3166af640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-09-12T06:34:09.066 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:34:09.068+0000 7fa315eae640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-09-12T06:34:09.067 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:34:09.069+0000 7fa314eac640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-12T06:34:09.119 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:34:09.121+0000 7fa310ea4640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-09-12T06:34:09.120 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:34:09.122+0000 7fa311ea6640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-12T06:34:09.326 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:34:09.328+0000 7f80bc980640 -1 bdev(0x556b0184ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 3 2024-09-12T06:34:09.554 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T06:34:09.562 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:34:09.779 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:34:09.897 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:34:09.898 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror test snap6 2024-09-12T06:34:09.898 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:34:09.898 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:34:09.898 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap6 2024-09-12T06:34:09.898 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:34:09.898 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export 2024-09-12T06:34:09.898 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:34:09.898 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:34:09.898 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export 2024-09-12T06:34:10.410 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T06:34:10.421 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:34:10.578 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:34:10.580+0000 7fa3166af640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-12T06:34:10.702 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:34:10.704+0000 7fa3136a9640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-12T06:34:10.703 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:34:10.705+0000 7fa314eac640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-12T06:34:10.706 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:34:10.708+0000 7fa3176b1640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-12T06:34:10.827 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:34:10.829+0000 7fa311ea6640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-12T06:34:10.828 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:34:10.830+0000 7fa317eb2640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-09-12T06:34:11.029 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:34:11.031+0000 7fa317eb2640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-12T06:34:11.153 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T06:34:11.163 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:34:11.374 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:34:11.490 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:34:11.490 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ clean_snap_name=snap6-clean 2024-09-12T06:34:11.490 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_snap cluster2 mirror test snap6-clean 2024-09-12T06:34:11.490 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:34:11.490 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:34:11.490 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:34:11.491 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap6-clean 2024-09-12T06:34:11.491 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6-clean --debug-rbd=20 --debug-journaler=20 2024-09-12T06:34:11.707 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 10` 2024-09-12T06:34:11.707 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ stress_write_image cluster2 mirror test 2024-09-12T06:34:11.707 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:34:11.707 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:34:11.707 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:34:11.707 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T06:34:11.708 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local duration=7 2024-09-12T06:34:11.708 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set +e 2024-09-12T06:34:11.708 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-12T06:34:14.087 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:100 IOs, 68563968 bytes 2024-09-12T06:34:16.578 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:34:16.580+0000 7fac12a52640 -1 reset not still connected to 0x55749c9a7860 2024-09-12T06:34:17.716 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:200 IOs, 125191168 bytes 2024-09-12T06:34:18.720 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ error_code=124 2024-09-12T06:34:18.721 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set -e 2024-09-12T06:34:18.721 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T06:34:18.721 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:34:18.721 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-12T06:34:18.721 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:34:18.721 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:34:18.721 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:34:18.721 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state_pattern=up+replaying 2024-09-12T06:34:18.721 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local description_pattern= 2024-09-12T06:34:18.721 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local service_pattern= 2024-09-12T06:34:18.721 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:34:18.721 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:34:19.723 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-12T06:34:19.723 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:34:19.723 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:34:19.723 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:34:19.723 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state_pattern=up+replaying 2024-09-12T06:34:19.723 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local description_pattern= 2024-09-12T06:34:19.723 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local service_pattern= 2024-09-12T06:34:19.724 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local status 2024-09-12T06:34:19.724 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ CEPH_ARGS= 2024-09-12T06:34:19.724 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-12T06:34:19.778 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ status='test: 2024-09-12T06:34:19.779 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: global_id: 5bf4c9df-a9e9-4aaa-9110-4727ba332309 2024-09-12T06:34:19.779 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: state: up+replaying 2024-09-12T06:34:19.779 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: description: replaying, {"bytes_per_second":3204991.17,"entries_behind_primary":0,"entries_per_second":200.47,"non_primary_position":{"entry_tid":1,"object_number":81,"tag_tid":18},"primary_position":{"entry_tid":1,"object_number":81,"tag_tid":18}} 2024-09-12T06:34:19.779 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: service: mirror.0 on smithi152 2024-09-12T06:34:19.779 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: last_update: 2024-09-12 06:34:09 2024-09-12T06:34:19.779 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: peer_sites: 2024-09-12T06:34:19.779 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: name: cluster2 2024-09-12T06:34:19.779 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: state: up+stopped 2024-09-12T06:34:19.779 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: description: local image is primary 2024-09-12T06:34:19.779 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: last_update: 2024-09-12 06:34:19' 2024-09-12T06:34:19.779 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep '^ state: .*up+replaying' 2024-09-12T06:34:19.780 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: state: up+replaying 2024-09-12T06:34:19.780 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep '^ description: .*' 2024-09-12T06:34:19.781 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: description: replaying, {"bytes_per_second":3204991.17,"entries_behind_primary":0,"entries_per_second":200.47,"non_primary_position":{"entry_tid":1,"object_number":81,"tag_tid":18},"primary_position":{"entry_tid":1,"object_number":81,"tag_tid":18}} 2024-09-12T06:34:19.781 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n '' ']' 2024-09-12T06:34:19.781 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo up+replaying 2024-09-12T06:34:19.781 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep '^up+' 2024-09-12T06:34:19.782 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:up+replaying 2024-09-12T06:34:19.782 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep 'service: *mirror..* on ' 2024-09-12T06:34:19.782 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: service: mirror.0 on smithi152 2024-09-12T06:34:19.783 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local last_update 2024-09-12T06:34:19.783 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-12T06:34:19.786 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ last_update='2024-09-12 06:34:09' 2024-09-12T06:34:19.786 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-12 06:34:09' 2024-09-12T06:34:19.786 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:34:19.786 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:34:19.786 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:34:19.786 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state_pattern=up+replaying 2024-09-12T06:34:19.786 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'prev_last_update=2024-09-12 06:34:09' 2024-09-12T06:34:19.787 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local status 2024-09-12T06:34:19.787 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ CEPH_ARGS= 2024-09-12T06:34:19.787 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-12T06:34:19.846 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ status='OKOKOK142614264mirror.0smithi15219.3.0-4883-gd61b0886trueOK42684271mirror.1smithi15219.3.0-4883-gd61b0886falseOKtest5bf4c9df-a9e9-4aaa-9110-4727ba332309up+replayingreplaying, {"bytes_per_second":3204991.17,"entries_behind_primary":0,"entries_per_second":200.47,"non_primary_position":{"entry_tid":1,"object_number":81,"tag_tid":18},"primary_position":{"entry_tid":1,"object_number":81,"tag_tid":18}}42614264mirror.0smithi1522024-09-12 06:34:09cluster24b9b47ae-ca94-4dab-91d1-08bce6a93349up+stoppedlocal image is primary2024-09-12 06:34:19' 2024-09-12T06:34:19.846 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local last_update state 2024-09-12T06:34:19.846 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-12T06:34:19.848 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ last_update='2024-09-12 06:34:09' 2024-09-12T06:34:19.848 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-12T06:34:19.850 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ state=up+replaying 2024-09-12T06:34:19.850 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo up+replaying 2024-09-12T06:34:19.850 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep up+replaying 2024-09-12T06:34:19.850 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:up+replaying 2024-09-12T06:34:19.851 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:34:19.851 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:34:19.852 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ snap_name=snap7 2024-09-12T06:34:19.852 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_snap cluster2 mirror test snap7 2024-09-12T06:34:19.852 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:34:19.852 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:34:19.852 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:34:19.852 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap7 2024-09-12T06:34:19.852 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7 --debug-rbd=20 --debug-journaler=20 2024-09-12T06:34:49.786 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-12T06:34:49.786 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:34:49.786 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:34:49.786 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:34:49.786 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-12T06:34:49.786 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:34:49.787 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:34:49.787 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:34:49.787 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=started 2024-09-12T06:34:49.787 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:34:49.787 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:34:49.787 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:34:50.789 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-12T06:34:50.789 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:34:50.789 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:34:50.789 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:34:50.790 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=started 2024-09-12T06:34:50.790 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local status_result 2024-09-12T06:34:50.790 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=stopped 2024-09-12T06:34:50.790 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-12T06:34:50.790 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_instance=cluster1 2024-09-12T06:34:50.790 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -i state 2024-09-12T06:34:50.791 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:34:50.791 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:34:50.791 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance=cluster1 2024-09-12T06:34:50.791 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local loop_instance 2024-09-12T06:34:50.791 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:34:50.791 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sleep 0 2024-09-12T06:34:50.792 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:34:50.792 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ seq 0 1 2024-09-12T06:34:50.798 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:34:50.798 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-12T06:34:50.798 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:0 2024-09-12T06:34:50.799 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:34:50.799 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:34:50.799 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:34:50.799 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:34:50.799 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:34:50.800 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:34:50.800 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:34:50.800 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:34:50.800 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:34:50.800 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:34:50.801 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:34:50.801 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:34:50.801 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:34:50.801 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:34:50.802 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:0 2024-09-12T06:34:50.802 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:34:50.802 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:34:50.802 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:34:50.802 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:0 2024-09-12T06:34:50.802 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:34:50.803 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:34:50.803 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:34:50.803 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:34:50.803 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T06:34:50.803 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:34:50.803 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:34:50.804 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=0 2024-09-12T06:34:50.804 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=0 2024-09-12T06:34:50.804 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:34:50.814 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:34:50.814 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:34:50.814 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:34:50.814 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-12T06:34:50.902 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ return 0 2024-09-12T06:34:50.903 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T06:34:50.903 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo ' "state": "Replaying"' 2024-09-12T06:34:50.903 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep -i Replaying 2024-09-12T06:34:50.904 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: "state": "Replaying" 2024-09-12T06:34:50.905 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_state=started 2024-09-12T06:34:50.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test started = started 2024-09-12T06:34:50.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:34:50.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror test 2024-09-12T06:34:50.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:34:50.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:34:50.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:34:50.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:34:50.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:34:50.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' journal = journal ']' 2024-09-12T06:34:50.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror test 2024-09-12T06:34:50.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:34:50.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:34:50.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:34:50.907 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:34:50.907 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:34:50.907 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-12T06:34:50.907 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-12T06:34:50.907 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:34:50.907 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:34:50.907 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:34:51.106 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror test 2024-09-12T06:34:51.106 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:34:51.107 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:34:51.107 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:34:51.107 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:34:51.107 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n test ']' 2024-09-12T06:34:51.107 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-12T06:34:51.107 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-12T06:34:51.107 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:34:51.107 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:34:51.107 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:34:51.107 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:34:51.107 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:34:51.107 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:34:51.107 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:34:51.108 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:34:51.108 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:34:51.110 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:34:51.110 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-12T06:34:51.110 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:34:51.111 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:34:51.111 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:34:51.111 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:34:51.111 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:34:51.111 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:34:51.111 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:34:51.111 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:34:51.111 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:34:51.111 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:34:51.111 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:34:51.111 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:34:51.111 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:34:51.111 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:34:51.111 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:34:51.112 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:34:51.112 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:34:51.112 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:34:51.112 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:34:51.112 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:34:51.112 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:34:51.112 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:34:51.112 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:34:51.112 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:34:51.112 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:34:51.112 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:34:51.112 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:34:51.112 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:34:51.112 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:34:51.113 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:34:51.129 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:34:51.129 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:34:51.129 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:34:51.129 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-12T06:34:51.215 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:34:51.216 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-12T06:34:51.216 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:34:51.216 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:34:51.216 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:34:51.216 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-12T06:34:51.216 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:34:51.216 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:34:51.216 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:34:51.216 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:34:51.216 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-12T06:34:51.216 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-test.status 2024-09-12T06:34:51.216 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:34:51.217 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:34:51.217 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-12T06:34:51.217 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:34:51.263 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 23 2024-09-12T06:34:51.263 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 23 2024-09-12T06:34:51.263 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:34:51.263 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=21, entry_tid=1], [object_number=92, tag_tid=21, entry_tid=0], [object_number=95, tag_tid=20, entry_tid=9535], [object_number=94, tag_tid=20, entry_tid=9534]]], state=connected] 2024-09-12T06:34:51.263 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=93, tag_tid=21, entry_tid=1], [object_number=92, tag_tid=21, entry_tid=0], [object_number=95, tag_tid=20, entry_tid=9535], [object_number=94, tag_tid=20, entry_tid=9534]]], state=connected] 2024-09-12T06:34:51.266 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:34:51.267 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=93, tag_tid=21, entry_tid=1' 2024-09-12T06:34:51.268 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-12T06:34:51.269 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:34:51.269 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:34:51.269 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:34:51.269 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-12T06:34:51.269 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:34:51.269 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:34:51.269 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:34:51.269 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:34:51.269 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-12T06:34:51.269 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-test.status 2024-09-12T06:34:51.269 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:34:51.270 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:34:51.270 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-12T06:34:51.270 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:34:51.316 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 23 2024-09-12T06:34:51.316 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 23 2024-09-12T06:34:51.316 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:34:51.316 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=21, entry_tid=1], [object_number=92, tag_tid=21, entry_tid=0], [object_number=95, tag_tid=20, entry_tid=9535], [object_number=94, tag_tid=20, entry_tid=9534]]], state=connected] 2024-09-12T06:34:51.316 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=93, tag_tid=21, entry_tid=1], [object_number=92, tag_tid=21, entry_tid=0], [object_number=95, tag_tid=20, entry_tid=9535], [object_number=94, tag_tid=20, entry_tid=9534]]], state=connected] 2024-09-12T06:34:51.318 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:34:51.321 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=93, tag_tid=21, entry_tid=1' 2024-09-12T06:34:51.321 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=93, tag_tid=21, entry_tid=1' -a 'object_number=93, tag_tid=21, entry_tid=1' = 'object_number=93, tag_tid=21, entry_tid=1' 2024-09-12T06:34:51.321 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:34:51.321 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_snap_present cluster1 mirror test snap7 2024-09-12T06:34:51.321 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:34:51.321 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:34:51.321 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:34:51.321 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap7 2024-09-12T06:34:51.321 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:34:51.322 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T06:34:51.322 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:34:52.322 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 info mirror/test@snap7 2024-09-12T06:34:52.378 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:rbd image 'test': 2024-09-12T06:34:52.378 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: size 512 MiB in 128 objects 2024-09-12T06:34:52.378 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: order 22 (4 MiB objects) 2024-09-12T06:34:52.378 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: snapshot_count: 13 2024-09-12T06:34:52.378 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: id: 10a8e43c53b8 2024-09-12T06:34:52.378 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: block_name_prefix: rbd_data.10a8e43c53b8 2024-09-12T06:34:52.378 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: format: 2 2024-09-12T06:34:52.379 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: features: layering, exclusive-lock, journaling 2024-09-12T06:34:52.379 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: op_features: 2024-09-12T06:34:52.379 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: flags: 2024-09-12T06:34:52.379 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: create_timestamp: Thu Sep 12 06:29:41 2024 2024-09-12T06:34:52.379 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: access_timestamp: Thu Sep 12 06:34:08 2024 2024-09-12T06:34:52.379 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: modify_timestamp: Thu Sep 12 06:34:13 2024 2024-09-12T06:34:52.379 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: protected: False 2024-09-12T06:34:52.379 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: journal: 10a8e43c53b8 2024-09-12T06:34:52.379 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring state: enabled 2024-09-12T06:34:52.379 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring mode: journal 2024-09-12T06:34:52.379 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring global id: 5bf4c9df-a9e9-4aaa-9110-4727ba332309 2024-09-12T06:34:52.379 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring primary: false 2024-09-12T06:34:52.382 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:34:52.382 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n snap6-clean ']' 2024-09-12T06:34:52.382 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror test snap6-clean 2024-09-12T06:34:52.382 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:34:52.383 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:34:52.383 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap6-clean 2024-09-12T06:34:52.383 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:34:52.383 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export 2024-09-12T06:34:52.383 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:34:52.383 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:34:52.383 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export 2024-09-12T06:34:52.898 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T06:34:52.908 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:34:53.277 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:34:53.279+0000 7fa314eac640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-12T06:34:53.280 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:34:53.282+0000 7fa3136a9640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-12T06:34:53.280 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:34:53.282+0000 7fa315eae640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-12T06:34:53.799 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T06:34:53.809 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:34:54.019 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:34:54.134 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:34:54.134 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror test snap7 2024-09-12T06:34:54.134 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:34:54.134 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:34:54.134 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap7 2024-09-12T06:34:54.134 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:34:54.134 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export 2024-09-12T06:34:54.134 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:34:54.134 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:34:54.134 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export 2024-09-12T06:34:54.637 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T06:34:54.647 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:34:55.106 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:34:55.107+0000 7fa315eae640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-12T06:34:55.108 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:34:55.109+0000 7fa314eac640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-09-12T06:34:55.108 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:34:55.109+0000 7fa3166af640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-12T06:34:55.108 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:34:55.110+0000 7fa3186b3640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-12T06:34:55.248 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:34:55.249+0000 7fa3126a7640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-09-12T06:34:55.248 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:34:55.250+0000 7fa312ea8640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-09-12T06:34:55.410 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:34:55.412+0000 7fabfe618640 -1 bdev(0x55749a596c00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-09-12T06:34:55.688 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T06:34:55.699 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:34:55.911 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:34:56.025 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:34:56.026 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ clean_snap_name=snap7-clean 2024-09-12T06:34:56.026 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_snap cluster2 mirror test snap7-clean 2024-09-12T06:34:56.026 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:34:56.026 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:34:56.026 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:34:56.026 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap7-clean 2024-09-12T06:34:56.026 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7-clean --debug-rbd=20 --debug-journaler=20 2024-09-12T06:34:56.802 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 10` 2024-09-12T06:34:56.802 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ stress_write_image cluster2 mirror test 2024-09-12T06:34:56.802 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:34:56.802 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:34:56.802 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:34:56.802 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T06:34:56.803 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local duration=8 2024-09-12T06:34:56.803 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set +e 2024-09-12T06:34:56.804 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-12T06:34:59.534 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:100 IOs, 78226432 bytes 2024-09-12T06:35:01.831 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:200 IOs, 139158528 bytes 2024-09-12T06:35:03.634 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:300 IOs, 167932928 bytes 2024-09-12T06:35:04.815 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ error_code=124 2024-09-12T06:35:04.815 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set -e 2024-09-12T06:35:04.815 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T06:35:04.815 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:35:04.815 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-12T06:35:04.815 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:35:04.816 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:35:04.816 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:35:04.816 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state_pattern=up+replaying 2024-09-12T06:35:04.816 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local description_pattern= 2024-09-12T06:35:04.816 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local service_pattern= 2024-09-12T06:35:04.816 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:35:04.816 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:35:05.816 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-12T06:35:05.816 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:35:05.816 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:35:05.816 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:35:05.816 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state_pattern=up+replaying 2024-09-12T06:35:05.817 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local description_pattern= 2024-09-12T06:35:05.817 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local service_pattern= 2024-09-12T06:35:05.817 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local status 2024-09-12T06:35:05.817 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ CEPH_ARGS= 2024-09-12T06:35:05.817 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-12T06:35:05.874 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ status='test: 2024-09-12T06:35:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: global_id: 5bf4c9df-a9e9-4aaa-9110-4727ba332309 2024-09-12T06:35:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: state: up+replaying 2024-09-12T06:35:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: description: replaying, {"bytes_per_second":3782254.0,"entries_behind_primary":0,"entries_per_second":234.83,"non_primary_position":{"entry_tid":5567,"object_number":91,"tag_tid":20},"primary_position":{"entry_tid":5338,"object_number":90,"tag_tid":20}} 2024-09-12T06:35:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: service: mirror.0 on smithi152 2024-09-12T06:35:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: last_update: 2024-09-12 06:34:39 2024-09-12T06:35:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: peer_sites: 2024-09-12T06:35:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: name: cluster2 2024-09-12T06:35:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: state: up+stopped 2024-09-12T06:35:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: description: local image is primary 2024-09-12T06:35:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: last_update: 2024-09-12 06:35:04' 2024-09-12T06:35:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep '^ state: .*up+replaying' 2024-09-12T06:35:05.876 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: state: up+replaying 2024-09-12T06:35:05.876 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep '^ description: .*' 2024-09-12T06:35:05.876 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: description: replaying, {"bytes_per_second":3782254.0,"entries_behind_primary":0,"entries_per_second":234.83,"non_primary_position":{"entry_tid":5567,"object_number":91,"tag_tid":20},"primary_position":{"entry_tid":5338,"object_number":90,"tag_tid":20}} 2024-09-12T06:35:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n '' ']' 2024-09-12T06:35:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo up+replaying 2024-09-12T06:35:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep '^up+' 2024-09-12T06:35:05.878 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:up+replaying 2024-09-12T06:35:05.878 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep 'service: *mirror..* on ' 2024-09-12T06:35:05.878 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: service: mirror.0 on smithi152 2024-09-12T06:35:05.879 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local last_update 2024-09-12T06:35:05.879 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-12T06:35:05.881 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ last_update='2024-09-12 06:34:39' 2024-09-12T06:35:05.881 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-12 06:34:39' 2024-09-12T06:35:05.881 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:35:05.881 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:35:05.881 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:35:05.881 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state_pattern=up+replaying 2024-09-12T06:35:05.881 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'prev_last_update=2024-09-12 06:34:39' 2024-09-12T06:35:05.882 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local status 2024-09-12T06:35:05.882 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ CEPH_ARGS= 2024-09-12T06:35:05.882 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-12T06:35:05.943 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ status='OKOKOK142614264mirror.0smithi15219.3.0-4883-gd61b0886trueOK42684271mirror.1smithi15219.3.0-4883-gd61b0886falseOKtest5bf4c9df-a9e9-4aaa-9110-4727ba332309up+replayingreplaying, {"bytes_per_second":3782254.0,"entries_behind_primary":0,"entries_per_second":234.83,"non_primary_position":{"entry_tid":5567,"object_number":91,"tag_tid":20},"primary_position":{"entry_tid":5338,"object_number":90,"tag_tid":20}}42614264mirror.0smithi1522024-09-12 06:34:39cluster24b9b47ae-ca94-4dab-91d1-08bce6a93349up+stoppedlocal image is primary2024-09-12 06:35:04' 2024-09-12T06:35:05.943 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local last_update state 2024-09-12T06:35:05.943 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-12T06:35:05.945 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ last_update='2024-09-12 06:34:39' 2024-09-12T06:35:05.945 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-12T06:35:05.947 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ state=up+replaying 2024-09-12T06:35:05.947 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo up+replaying 2024-09-12T06:35:05.947 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep up+replaying 2024-09-12T06:35:05.947 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:up+replaying 2024-09-12T06:35:05.948 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:35:05.948 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:35:05.948 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ snap_name=snap8 2024-09-12T06:35:05.949 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_snap cluster2 mirror test snap8 2024-09-12T06:35:05.949 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:35:05.949 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:35:05.949 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:35:05.949 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap8 2024-09-12T06:35:05.949 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8 --debug-rbd=20 --debug-journaler=20 2024-09-12T06:35:06.532 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:35:06.534+0000 7fac12a52640 -1 reset not still connected to 0x55749c9a7860 2024-09-12T06:35:35.847 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-12T06:35:35.848 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:35:35.848 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:35:35.848 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:35:35.848 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-12T06:35:35.848 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:35:35.848 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:35:35.848 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:35:35.848 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=started 2024-09-12T06:35:35.848 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:35:35.848 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:35:35.849 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:35:36.848 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-12T06:35:36.848 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:35:36.848 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:35:36.848 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:35:36.849 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=started 2024-09-12T06:35:36.849 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local status_result 2024-09-12T06:35:36.849 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=stopped 2024-09-12T06:35:36.850 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-12T06:35:36.850 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_instance=cluster1 2024-09-12T06:35:36.850 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -i state 2024-09-12T06:35:36.850 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:35:36.850 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:35:36.850 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance=cluster1 2024-09-12T06:35:36.850 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local loop_instance 2024-09-12T06:35:36.850 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:35:36.850 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sleep 0 2024-09-12T06:35:36.851 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:35:36.851 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ seq 0 1 2024-09-12T06:35:36.852 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:35:36.853 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-12T06:35:36.853 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:0 2024-09-12T06:35:36.853 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:35:36.853 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:35:36.853 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:35:36.853 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:35:36.853 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:35:36.853 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:35:36.853 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:35:36.853 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:35:36.853 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:35:36.853 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:35:36.853 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:35:36.854 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:35:36.854 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:35:36.854 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:35:36.854 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:0 2024-09-12T06:35:36.855 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:35:36.855 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:35:36.856 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:35:36.856 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:0 2024-09-12T06:35:36.856 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:35:36.856 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:35:36.856 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:35:36.856 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:35:36.856 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T06:35:36.856 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:35:36.856 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:35:36.856 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=0 2024-09-12T06:35:36.856 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=0 2024-09-12T06:35:36.856 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:35:36.874 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:35:36.874 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:35:36.874 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:35:36.874 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-12T06:35:36.959 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ return 0 2024-09-12T06:35:36.959 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T06:35:36.960 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo ' "state": "Replaying"' 2024-09-12T06:35:36.960 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep -i Replaying 2024-09-12T06:35:36.961 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: "state": "Replaying" 2024-09-12T06:35:36.962 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_state=started 2024-09-12T06:35:36.962 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test started = started 2024-09-12T06:35:36.962 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:35:36.962 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror test 2024-09-12T06:35:36.962 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:35:36.962 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:35:36.962 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:35:36.963 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:35:36.963 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:35:36.963 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' journal = journal ']' 2024-09-12T06:35:36.963 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror test 2024-09-12T06:35:36.963 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:35:36.963 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:35:36.963 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:35:36.963 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:35:36.963 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:35:36.963 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-12T06:35:36.963 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-12T06:35:36.963 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:35:36.963 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:35:36.963 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:35:37.163 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror test 2024-09-12T06:35:37.163 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:35:37.163 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:35:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:35:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:35:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n test ']' 2024-09-12T06:35:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-12T06:35:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-12T06:35:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:35:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:35:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:35:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:35:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:35:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:35:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:35:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:35:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:35:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:35:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-12T06:35:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:35:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:35:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:35:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:35:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:35:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:35:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:35:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:35:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:35:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:35:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:35:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:35:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:35:37.169 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:35:37.169 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:35:37.169 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:35:37.169 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:35:37.169 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:35:37.169 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:35:37.169 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:35:37.169 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:35:37.169 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:35:37.169 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:35:37.169 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:35:37.169 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:35:37.169 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:35:37.169 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:35:37.170 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:35:37.170 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:35:37.170 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:35:37.185 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:35:37.185 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:35:37.185 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:35:37.185 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-12T06:35:37.267 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:35:37.269 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-12T06:35:37.269 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:35:37.269 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:35:37.269 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:35:37.269 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-12T06:35:37.269 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:35:37.269 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:35:37.269 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:35:37.269 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:35:37.269 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-12T06:35:37.269 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-test.status 2024-09-12T06:35:37.270 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:35:37.270 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:35:37.270 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-12T06:35:37.270 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:35:37.317 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 27 2024-09-12T06:35:37.317 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 27 2024-09-12T06:35:37.318 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:35:37.318 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=109, tag_tid=24, entry_tid=1], [object_number=108, tag_tid=24, entry_tid=0], [object_number=111, tag_tid=23, entry_tid=12175], [object_number=110, tag_tid=23, entry_tid=12174]]], state=connected] 2024-09-12T06:35:37.318 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=108, tag_tid=23, entry_tid=12176], [object_number=111, tag_tid=23, entry_tid=12175], [object_number=110, tag_tid=23, entry_tid=12174], [object_number=109, tag_tid=23, entry_tid=12173]]], state=connected] 2024-09-12T06:35:37.320 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:35:37.321 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=109, tag_tid=24, entry_tid=1' 2024-09-12T06:35:37.322 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-12T06:35:37.323 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:35:37.323 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:35:37.323 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:35:37.323 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-12T06:35:37.323 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:35:37.323 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:35:37.323 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:35:37.323 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:35:37.323 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-12T06:35:37.323 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-test.status 2024-09-12T06:35:37.323 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:35:37.324 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:35:37.324 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-12T06:35:37.324 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:35:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 27 2024-09-12T06:35:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 27 2024-09-12T06:35:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:35:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=109, tag_tid=24, entry_tid=1], [object_number=108, tag_tid=24, entry_tid=0], [object_number=111, tag_tid=23, entry_tid=12175], [object_number=110, tag_tid=23, entry_tid=12174]]], state=connected] 2024-09-12T06:35:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=108, tag_tid=23, entry_tid=12176], [object_number=111, tag_tid=23, entry_tid=12175], [object_number=110, tag_tid=23, entry_tid=12174], [object_number=109, tag_tid=23, entry_tid=12173]]], state=connected] 2024-09-12T06:35:37.376 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:35:37.379 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=108, tag_tid=23, entry_tid=12176' 2024-09-12T06:35:37.379 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=109, tag_tid=24, entry_tid=1' -a 'object_number=109, tag_tid=24, entry_tid=1' = 'object_number=108, tag_tid=23, entry_tid=12176' 2024-09-12T06:35:37.379 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 'object_number=108, tag_tid=23, entry_tid=12176' '!=' '' 2024-09-12T06:35:37.379 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ break 2024-09-12T06:35:37.381 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 'object_number=108, tag_tid=23, entry_tid=12176' = '' 2024-09-12T06:35:37.381 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ last_mirror_pos='object_number=108, tag_tid=23, entry_tid=12176' 2024-09-12T06:35:37.382 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=109, tag_tid=24, entry_tid=1' 2024-09-12T06:35:37.383 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-12T06:35:37.383 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:35:37.384 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_tag=24 2024-09-12T06:35:37.385 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=108, tag_tid=23, entry_tid=12176' 2024-09-12T06:35:37.385 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-12T06:35:37.385 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:35:37.385 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_tag=23 2024-09-12T06:35:37.385 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=109, tag_tid=24, entry_tid=1' 2024-09-12T06:35:37.385 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-12T06:35:37.385 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:35:37.385 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_entry=1 2024-09-12T06:35:37.385 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=108, tag_tid=23, entry_tid=12176' 2024-09-12T06:35:37.385 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-12T06:35:37.385 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:35:37.385 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_entry=12176 2024-09-12T06:35:37.385 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 24 = 23 -a 1 -le 12176 2024-09-12T06:35:37.386 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:35:37.386 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:35:37.386 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:35:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror test 2024-09-12T06:35:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:35:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:35:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:35:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:35:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n test ']' 2024-09-12T06:35:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-12T06:35:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-12T06:35:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:35:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:35:37.588 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:35:37.588 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:35:37.588 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:35:37.588 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:35:37.588 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:35:37.588 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:35:37.588 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:35:37.591 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:35:37.591 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-12T06:35:37.591 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:35:37.591 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:35:37.591 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:35:37.591 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:35:37.591 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:35:37.592 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:35:37.592 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:35:37.592 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:35:37.592 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:35:37.592 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:35:37.592 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:35:37.592 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:35:37.592 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:35:37.592 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:35:37.592 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:35:37.592 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:35:37.592 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:35:37.592 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:35:37.593 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:35:37.593 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:35:37.593 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:35:37.593 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:35:37.593 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:35:37.593 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:35:37.593 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:35:37.593 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:35:37.593 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:35:37.593 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:35:37.593 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:35:37.593 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:35:37.608 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:35:37.608 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:35:37.608 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:35:37.608 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-12T06:35:37.689 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:35:37.689 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-12T06:35:37.689 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:35:37.689 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:35:37.689 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:35:37.689 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-12T06:35:37.690 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:35:37.690 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:35:37.690 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:35:37.690 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:35:37.690 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-12T06:35:37.690 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-test.status 2024-09-12T06:35:37.690 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:35:37.691 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:35:37.691 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-12T06:35:37.691 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:35:37.737 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 27 2024-09-12T06:35:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 27 2024-09-12T06:35:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:35:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=109, tag_tid=24, entry_tid=1], [object_number=108, tag_tid=24, entry_tid=0], [object_number=111, tag_tid=23, entry_tid=12175], [object_number=110, tag_tid=23, entry_tid=12174]]], state=connected] 2024-09-12T06:35:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=108, tag_tid=23, entry_tid=12176], [object_number=111, tag_tid=23, entry_tid=12175], [object_number=110, tag_tid=23, entry_tid=12174], [object_number=109, tag_tid=23, entry_tid=12173]]], state=connected] 2024-09-12T06:35:37.740 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:35:37.742 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=109, tag_tid=24, entry_tid=1' 2024-09-12T06:35:37.743 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-12T06:35:37.743 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:35:37.743 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:35:37.743 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:35:37.743 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-12T06:35:37.743 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:35:37.743 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:35:37.743 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:35:37.743 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:35:37.744 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-12T06:35:37.744 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-test.status 2024-09-12T06:35:37.744 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:35:37.744 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:35:37.744 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-12T06:35:37.745 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:35:37.792 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 27 2024-09-12T06:35:37.792 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 27 2024-09-12T06:35:37.792 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:35:37.792 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=109, tag_tid=24, entry_tid=1], [object_number=108, tag_tid=24, entry_tid=0], [object_number=111, tag_tid=23, entry_tid=12175], [object_number=110, tag_tid=23, entry_tid=12174]]], state=connected] 2024-09-12T06:35:37.792 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=108, tag_tid=23, entry_tid=12176], [object_number=111, tag_tid=23, entry_tid=12175], [object_number=110, tag_tid=23, entry_tid=12174], [object_number=109, tag_tid=23, entry_tid=12173]]], state=connected] 2024-09-12T06:35:37.795 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:35:37.797 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=108, tag_tid=23, entry_tid=12176' 2024-09-12T06:35:37.797 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=109, tag_tid=24, entry_tid=1' -a 'object_number=109, tag_tid=24, entry_tid=1' = 'object_number=108, tag_tid=23, entry_tid=12176' 2024-09-12T06:35:37.797 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 'object_number=108, tag_tid=23, entry_tid=12176' '!=' 'object_number=108, tag_tid=23, entry_tid=12176' 2024-09-12T06:35:37.797 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:35:37.797 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.4 2024-09-12T06:35:38.199 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror test 2024-09-12T06:35:38.199 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:35:38.199 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:35:38.199 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:35:38.199 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:35:38.199 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n test ']' 2024-09-12T06:35:38.200 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-12T06:35:38.200 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-12T06:35:38.200 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:35:38.200 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:35:38.200 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:35:38.200 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:35:38.200 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:35:38.200 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:35:38.200 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:35:38.200 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:35:38.201 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:35:38.203 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:35:38.203 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-12T06:35:38.204 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:35:38.204 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:35:38.204 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:35:38.204 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:35:38.204 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:35:38.204 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:35:38.204 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:35:38.204 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:35:38.204 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:35:38.204 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:35:38.204 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:35:38.204 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:35:38.205 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:35:38.205 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:35:38.205 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:35:38.205 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:35:38.205 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:35:38.205 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:35:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:35:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:35:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:35:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:35:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:35:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:35:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:35:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:35:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:35:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:35:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:35:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:35:38.221 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:35:38.221 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:35:38.221 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:35:38.221 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-12T06:35:38.304 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:35:38.305 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-12T06:35:38.305 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:35:38.305 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:35:38.305 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:35:38.305 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-12T06:35:38.305 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:35:38.305 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:35:38.305 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:35:38.305 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:35:38.305 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-12T06:35:38.305 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-test.status 2024-09-12T06:35:38.305 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:35:38.306 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:35:38.306 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-12T06:35:38.306 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:35:38.354 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 27 2024-09-12T06:35:38.355 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 27 2024-09-12T06:35:38.355 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:35:38.355 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=109, tag_tid=24, entry_tid=1], [object_number=108, tag_tid=24, entry_tid=0], [object_number=111, tag_tid=23, entry_tid=12175], [object_number=110, tag_tid=23, entry_tid=12174]]], state=connected] 2024-09-12T06:35:38.355 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=109, tag_tid=24, entry_tid=1], [object_number=108, tag_tid=24, entry_tid=0], [object_number=111, tag_tid=23, entry_tid=12175], [object_number=110, tag_tid=23, entry_tid=12174]]], state=connected] 2024-09-12T06:35:38.357 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:35:38.359 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=109, tag_tid=24, entry_tid=1' 2024-09-12T06:35:38.360 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-12T06:35:38.361 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:35:38.361 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:35:38.361 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:35:38.361 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-12T06:35:38.361 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:35:38.361 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:35:38.361 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:35:38.361 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:35:38.361 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-12T06:35:38.361 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-test.status 2024-09-12T06:35:38.362 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:35:38.362 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:35:38.363 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-12T06:35:38.363 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:35:38.410 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 27 2024-09-12T06:35:38.410 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 27 2024-09-12T06:35:38.410 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:35:38.411 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=109, tag_tid=24, entry_tid=1], [object_number=108, tag_tid=24, entry_tid=0], [object_number=111, tag_tid=23, entry_tid=12175], [object_number=110, tag_tid=23, entry_tid=12174]]], state=connected] 2024-09-12T06:35:38.411 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=109, tag_tid=24, entry_tid=1], [object_number=108, tag_tid=24, entry_tid=0], [object_number=111, tag_tid=23, entry_tid=12175], [object_number=110, tag_tid=23, entry_tid=12174]]], state=connected] 2024-09-12T06:35:38.413 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:35:38.416 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=109, tag_tid=24, entry_tid=1' 2024-09-12T06:35:38.416 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=109, tag_tid=24, entry_tid=1' -a 'object_number=109, tag_tid=24, entry_tid=1' = 'object_number=109, tag_tid=24, entry_tid=1' 2024-09-12T06:35:38.416 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:35:38.417 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_snap_present cluster1 mirror test snap8 2024-09-12T06:35:38.417 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:35:38.417 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:35:38.417 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:35:38.417 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap8 2024-09-12T06:35:38.418 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:35:38.418 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T06:35:38.418 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:35:39.416 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 info mirror/test@snap8 2024-09-12T06:35:39.470 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:rbd image 'test': 2024-09-12T06:35:39.470 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: size 512 MiB in 128 objects 2024-09-12T06:35:39.470 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: order 22 (4 MiB objects) 2024-09-12T06:35:39.470 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: snapshot_count: 15 2024-09-12T06:35:39.470 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: id: 10a8e43c53b8 2024-09-12T06:35:39.470 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: block_name_prefix: rbd_data.10a8e43c53b8 2024-09-12T06:35:39.471 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: format: 2 2024-09-12T06:35:39.471 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: features: layering, exclusive-lock, journaling 2024-09-12T06:35:39.471 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: op_features: 2024-09-12T06:35:39.471 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: flags: 2024-09-12T06:35:39.471 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: create_timestamp: Thu Sep 12 06:29:41 2024 2024-09-12T06:35:39.471 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: access_timestamp: Thu Sep 12 06:34:08 2024 2024-09-12T06:35:39.471 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: modify_timestamp: Thu Sep 12 06:35:14 2024 2024-09-12T06:35:39.471 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: protected: False 2024-09-12T06:35:39.471 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: journal: 10a8e43c53b8 2024-09-12T06:35:39.471 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring state: enabled 2024-09-12T06:35:39.471 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring mode: journal 2024-09-12T06:35:39.471 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring global id: 5bf4c9df-a9e9-4aaa-9110-4727ba332309 2024-09-12T06:35:39.471 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring primary: false 2024-09-12T06:35:39.474 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:35:39.474 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n snap7-clean ']' 2024-09-12T06:35:39.474 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror test snap7-clean 2024-09-12T06:35:39.474 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:35:39.474 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:35:39.474 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap7-clean 2024-09-12T06:35:39.475 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:35:39.475 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export 2024-09-12T06:35:39.475 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:35:39.475 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:35:39.475 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export 2024-09-12T06:35:40.007 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T06:35:40.017 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:35:40.224 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:35:40.225+0000 7fa315eae640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-12T06:35:40.224 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:35:40.225+0000 7fa310ea4640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-09-12T06:35:40.361 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:35:40.363+0000 7fa316eb0640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-12T06:35:40.362 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:35:40.363+0000 7fa311ea6640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-09-12T06:35:40.363 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:35:40.365+0000 7fa315eae640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-09-12T06:35:40.581 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:35:40.583+0000 7fa316eb0640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-12T06:35:40.707 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:35:40.709+0000 7fac01e1f640 -1 bdev(0x55749a596c00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 3 2024-09-12T06:35:40.778 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T06:35:40.788 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:35:40.999 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:35:41.114 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:35:41.114 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror test snap8 2024-09-12T06:35:41.114 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:35:41.114 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:35:41.114 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap8 2024-09-12T06:35:41.114 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:35:41.114 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export 2024-09-12T06:35:41.114 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:35:41.114 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:35:41.115 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export 2024-09-12T06:35:41.646 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T06:35:41.657 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:35:41.972 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:35:41.973+0000 7fa3156ad640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-09-12T06:35:42.104 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:35:42.106+0000 7fa311ea6640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-12T06:35:42.105 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:35:42.107+0000 7fa315eae640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-12T06:35:42.312 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:35:42.314+0000 7fa316eb0640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-12T06:35:42.493 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T06:35:42.502 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:35:42.713 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:35:42.828 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:35:42.828 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ clean_snap_name=snap8-clean 2024-09-12T06:35:42.828 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_snap cluster2 mirror test snap8-clean 2024-09-12T06:35:42.828 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:35:42.828 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:35:42.828 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:35:42.828 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap8-clean 2024-09-12T06:35:42.828 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8-clean --debug-rbd=20 --debug-journaler=20 2024-09-12T06:35:43.857 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 10` 2024-09-12T06:35:43.857 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ stress_write_image cluster2 mirror test 2024-09-12T06:35:43.857 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:35:43.857 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:35:43.857 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:35:43.857 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T06:35:43.859 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local duration=13 2024-09-12T06:35:43.859 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set +e 2024-09-12T06:35:43.859 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-12T06:35:46.495 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:100 IOs, 67982336 bytes 2024-09-12T06:35:48.793 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:200 IOs, 122956800 bytes 2024-09-12T06:35:51.402 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:300 IOs, 188960768 bytes 2024-09-12T06:35:52.345 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:400 IOs, 209409024 bytes 2024-09-12T06:35:53.303 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:500 IOs, 230791168 bytes 2024-09-12T06:35:54.981 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:600 IOs, 296725504 bytes 2024-09-12T06:35:55.901 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:700 IOs, 320770048 bytes 2024-09-12T06:35:56.867 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ error_code=124 2024-09-12T06:35:56.868 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set -e 2024-09-12T06:35:56.868 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T06:35:56.868 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:35:56.868 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-12T06:35:56.868 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:35:56.868 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:35:56.868 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:35:56.868 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state_pattern=up+replaying 2024-09-12T06:35:56.868 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local description_pattern= 2024-09-12T06:35:56.868 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local service_pattern= 2024-09-12T06:35:56.868 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:35:56.869 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:35:57.869 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-12T06:35:57.869 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:35:57.869 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:35:57.869 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:35:57.869 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state_pattern=up+replaying 2024-09-12T06:35:57.869 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local description_pattern= 2024-09-12T06:35:57.869 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local service_pattern= 2024-09-12T06:35:57.869 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local status 2024-09-12T06:35:57.869 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ CEPH_ARGS= 2024-09-12T06:35:57.869 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-12T06:35:57.933 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ status='test: 2024-09-12T06:35:57.933 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: global_id: 5bf4c9df-a9e9-4aaa-9110-4727ba332309 2024-09-12T06:35:57.933 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: state: up+replaying 2024-09-12T06:35:57.933 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: description: replaying, {"bytes_per_second":4452967.3,"entries_behind_primary":0,"entries_per_second":278.03,"non_primary_position":{"entry_tid":1,"object_number":109,"tag_tid":24},"primary_position":{"entry_tid":1,"object_number":109,"tag_tid":24}} 2024-09-12T06:35:57.933 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: service: mirror.0 on smithi152 2024-09-12T06:35:57.933 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: last_update: 2024-09-12 06:35:39 2024-09-12T06:35:57.934 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: peer_sites: 2024-09-12T06:35:57.934 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: name: cluster2 2024-09-12T06:35:57.934 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: state: up+stopped 2024-09-12T06:35:57.934 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: description: local image is primary 2024-09-12T06:35:57.934 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: last_update: 2024-09-12 06:35:54' 2024-09-12T06:35:57.934 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep '^ state: .*up+replaying' 2024-09-12T06:35:57.934 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: state: up+replaying 2024-09-12T06:35:57.935 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep '^ description: .*' 2024-09-12T06:35:57.935 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: description: replaying, {"bytes_per_second":4452967.3,"entries_behind_primary":0,"entries_per_second":278.03,"non_primary_position":{"entry_tid":1,"object_number":109,"tag_tid":24},"primary_position":{"entry_tid":1,"object_number":109,"tag_tid":24}} 2024-09-12T06:35:57.936 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n '' ']' 2024-09-12T06:35:57.936 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo up+replaying 2024-09-12T06:35:57.936 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep '^up+' 2024-09-12T06:35:57.936 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:up+replaying 2024-09-12T06:35:57.936 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep 'service: *mirror..* on ' 2024-09-12T06:35:57.937 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: service: mirror.0 on smithi152 2024-09-12T06:35:57.938 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local last_update 2024-09-12T06:35:57.938 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-12T06:35:57.939 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ last_update='2024-09-12 06:35:39' 2024-09-12T06:35:57.939 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-12 06:35:39' 2024-09-12T06:35:57.940 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:35:57.940 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:35:57.940 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:35:57.940 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state_pattern=up+replaying 2024-09-12T06:35:57.940 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'prev_last_update=2024-09-12 06:35:39' 2024-09-12T06:35:57.940 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local status 2024-09-12T06:35:57.940 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ CEPH_ARGS= 2024-09-12T06:35:57.940 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-12T06:35:58.001 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ status='OKOKOK142614264mirror.0smithi15219.3.0-4883-gd61b0886trueOK42684271mirror.1smithi15219.3.0-4883-gd61b0886falseOKtest5bf4c9df-a9e9-4aaa-9110-4727ba332309up+replayingreplaying, {"bytes_per_second":4452967.3,"entries_behind_primary":0,"entries_per_second":278.03,"non_primary_position":{"entry_tid":1,"object_number":109,"tag_tid":24},"primary_position":{"entry_tid":1,"object_number":109,"tag_tid":24}}42614264mirror.0smithi1522024-09-12 06:35:39cluster24b9b47ae-ca94-4dab-91d1-08bce6a93349up+stoppedlocal image is primary2024-09-12 06:35:54' 2024-09-12T06:35:58.002 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local last_update state 2024-09-12T06:35:58.002 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-12T06:35:58.003 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ last_update='2024-09-12 06:35:39' 2024-09-12T06:35:58.003 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-12T06:35:58.005 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ state=up+replaying 2024-09-12T06:35:58.005 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo up+replaying 2024-09-12T06:35:58.005 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep up+replaying 2024-09-12T06:35:58.006 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:up+replaying 2024-09-12T06:35:58.007 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:35:58.007 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:35:58.007 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ snap_name=snap9 2024-09-12T06:35:58.007 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_snap cluster2 mirror test snap9 2024-09-12T06:35:58.007 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:35:58.007 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:35:58.007 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:35:58.008 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap9 2024-09-12T06:35:58.008 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9 --debug-rbd=20 --debug-journaler=20 2024-09-12T06:36:27.938 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-12T06:36:27.939 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:36:27.939 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:36:27.939 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:36:27.939 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-12T06:36:27.939 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:36:27.939 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:36:27.939 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:36:27.939 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=started 2024-09-12T06:36:27.940 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:36:27.940 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:36:27.940 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:36:28.940 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-12T06:36:28.940 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:36:28.940 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:36:28.940 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:36:28.940 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=started 2024-09-12T06:36:28.940 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local status_result 2024-09-12T06:36:28.940 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=stopped 2024-09-12T06:36:28.940 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-12T06:36:28.940 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_instance=cluster1 2024-09-12T06:36:28.940 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:36:28.940 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:36:28.941 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -i state 2024-09-12T06:36:28.941 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance=cluster1 2024-09-12T06:36:28.941 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local loop_instance 2024-09-12T06:36:28.941 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:36:28.941 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sleep 0 2024-09-12T06:36:28.943 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:36:28.943 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ seq 0 1 2024-09-12T06:36:28.943 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:36:28.943 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-12T06:36:28.943 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:0 2024-09-12T06:36:28.943 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:36:28.943 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:36:28.943 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:36:28.943 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:36:28.944 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:36:28.944 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:36:28.944 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:36:28.944 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:36:28.944 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:36:28.944 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:36:28.944 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:36:28.944 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:36:28.944 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:36:28.944 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:36:28.946 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:0 2024-09-12T06:36:28.946 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:36:28.946 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:36:28.946 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:36:28.946 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:0 2024-09-12T06:36:28.946 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:36:28.946 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:36:28.946 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:36:28.946 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:36:28.946 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T06:36:28.946 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:36:28.946 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:36:28.946 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=0 2024-09-12T06:36:28.946 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=0 2024-09-12T06:36:28.947 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:36:28.962 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:36:28.962 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:36:28.962 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:36:28.962 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-12T06:36:29.044 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ return 0 2024-09-12T06:36:29.045 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T06:36:29.045 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo ' "state": "Replaying"' 2024-09-12T06:36:29.045 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep -i Replaying 2024-09-12T06:36:29.046 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: "state": "Replaying" 2024-09-12T06:36:29.047 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_state=started 2024-09-12T06:36:29.048 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test started = started 2024-09-12T06:36:29.048 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:36:29.048 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror test 2024-09-12T06:36:29.048 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:36:29.048 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:36:29.048 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:36:29.048 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:36:29.048 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:36:29.048 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' journal = journal ']' 2024-09-12T06:36:29.048 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror test 2024-09-12T06:36:29.048 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:36:29.048 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:36:29.048 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:36:29.049 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:36:29.049 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:36:29.049 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-12T06:36:29.049 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-12T06:36:29.049 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:36:29.049 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:36:29.049 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:36:29.249 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror test 2024-09-12T06:36:29.249 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:36:29.249 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:36:29.249 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:36:29.249 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:36:29.249 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n test ']' 2024-09-12T06:36:29.249 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-12T06:36:29.249 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-12T06:36:29.249 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:36:29.249 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:36:29.250 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:36:29.250 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:36:29.250 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:36:29.250 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:36:29.250 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:36:29.250 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:36:29.250 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:36:29.252 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:36:29.252 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-12T06:36:29.252 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:36:29.253 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:36:29.253 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:36:29.253 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:36:29.253 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:36:29.253 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:36:29.253 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:36:29.253 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:36:29.253 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:36:29.253 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:36:29.253 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:36:29.253 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:36:29.253 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:36:29.253 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:36:29.253 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:36:29.253 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:36:29.254 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:36:29.254 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:36:29.254 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:36:29.254 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:36:29.254 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:36:29.254 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:36:29.254 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:36:29.254 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:36:29.254 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:36:29.254 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:36:29.254 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:36:29.254 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:36:29.254 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:36:29.254 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:36:29.271 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:36:29.271 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:36:29.271 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:36:29.271 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-12T06:36:29.373 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:36:29.375 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-12T06:36:29.375 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:36:29.375 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:36:29.375 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:36:29.375 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-12T06:36:29.375 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:36:29.375 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:36:29.375 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:36:29.375 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:36:29.375 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-12T06:36:29.375 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-test.status 2024-09-12T06:36:29.375 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:36:29.376 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:29.377 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:29.377 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-12T06:36:29.436 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 32 2024-09-12T06:36:29.437 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 32 2024-09-12T06:36:29.437 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:36:29.437 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=129, tag_tid=27, entry_tid=1], [object_number=128, tag_tid=27, entry_tid=0], [object_number=131, tag_tid=26, entry_tid=19727], [object_number=130, tag_tid=26, entry_tid=19726]]], state=connected] 2024-09-12T06:36:29.437 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=131, tag_tid=26, entry_tid=16703], [object_number=130, tag_tid=26, entry_tid=16702], [object_number=129, tag_tid=26, entry_tid=16701], [object_number=128, tag_tid=26, entry_tid=16700]]], state=connected] 2024-09-12T06:36:29.439 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:29.441 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=129, tag_tid=27, entry_tid=1' 2024-09-12T06:36:29.442 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-12T06:36:29.442 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:36:29.442 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:36:29.442 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:36:29.442 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-12T06:36:29.442 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:36:29.442 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:36:29.443 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:36:29.443 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:36:29.443 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-12T06:36:29.443 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-test.status 2024-09-12T06:36:29.443 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:36:29.444 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:29.444 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-12T06:36:29.444 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:29.492 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 32 2024-09-12T06:36:29.492 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 32 2024-09-12T06:36:29.492 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:36:29.492 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=129, tag_tid=27, entry_tid=1], [object_number=128, tag_tid=27, entry_tid=0], [object_number=131, tag_tid=26, entry_tid=19727], [object_number=130, tag_tid=26, entry_tid=19726]]], state=connected] 2024-09-12T06:36:29.492 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=131, tag_tid=26, entry_tid=16703], [object_number=130, tag_tid=26, entry_tid=16702], [object_number=129, tag_tid=26, entry_tid=16701], [object_number=128, tag_tid=26, entry_tid=16700]]], state=connected] 2024-09-12T06:36:29.495 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:29.497 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=131, tag_tid=26, entry_tid=16703' 2024-09-12T06:36:29.497 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=129, tag_tid=27, entry_tid=1' -a 'object_number=129, tag_tid=27, entry_tid=1' = 'object_number=131, tag_tid=26, entry_tid=16703' 2024-09-12T06:36:29.497 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 'object_number=131, tag_tid=26, entry_tid=16703' '!=' '' 2024-09-12T06:36:29.497 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ break 2024-09-12T06:36:29.497 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 'object_number=131, tag_tid=26, entry_tid=16703' = '' 2024-09-12T06:36:29.497 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ last_mirror_pos='object_number=131, tag_tid=26, entry_tid=16703' 2024-09-12T06:36:29.497 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=129, tag_tid=27, entry_tid=1' 2024-09-12T06:36:29.497 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-12T06:36:29.497 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:36:29.498 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_tag=27 2024-09-12T06:36:29.498 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=131, tag_tid=26, entry_tid=16703' 2024-09-12T06:36:29.498 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-12T06:36:29.498 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:36:29.499 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_tag=26 2024-09-12T06:36:29.500 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=129, tag_tid=27, entry_tid=1' 2024-09-12T06:36:29.500 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-12T06:36:29.500 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:36:29.501 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_entry=1 2024-09-12T06:36:29.502 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=131, tag_tid=26, entry_tid=16703' 2024-09-12T06:36:29.502 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-12T06:36:29.502 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:36:29.503 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_entry=16703 2024-09-12T06:36:29.503 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 27 = 26 -a 1 -le 16703 2024-09-12T06:36:29.503 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:36:29.503 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:36:29.503 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:36:29.705 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror test 2024-09-12T06:36:29.705 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:36:29.705 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:36:29.705 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:36:29.705 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:36:29.705 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n test ']' 2024-09-12T06:36:29.706 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-12T06:36:29.706 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-12T06:36:29.706 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:36:29.706 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:36:29.706 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:36:29.706 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:36:29.706 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:36:29.706 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:36:29.706 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:36:29.706 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:36:29.706 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:36:29.709 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:36:29.709 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-12T06:36:29.709 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:36:29.709 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:36:29.709 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:36:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:36:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:36:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:36:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:36:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:36:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:36:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:36:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:36:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:36:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:36:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:36:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:36:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:36:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:36:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:36:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:36:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:36:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:36:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:36:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:36:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:36:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:36:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:36:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:36:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:36:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:36:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:36:29.727 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:36:29.727 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:36:29.727 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:36:29.727 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-12T06:36:29.817 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:36:29.818 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-12T06:36:29.818 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:36:29.818 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:36:29.818 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:36:29.818 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-12T06:36:29.818 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:36:29.818 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:36:29.818 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:36:29.818 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:36:29.818 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-12T06:36:29.818 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-test.status 2024-09-12T06:36:29.819 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:36:29.819 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:29.819 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-12T06:36:29.819 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:29.868 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 32 2024-09-12T06:36:29.868 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 32 2024-09-12T06:36:29.868 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:36:29.868 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=129, tag_tid=27, entry_tid=1], [object_number=128, tag_tid=27, entry_tid=0], [object_number=131, tag_tid=26, entry_tid=19727], [object_number=130, tag_tid=26, entry_tid=19726]]], state=connected] 2024-09-12T06:36:29.868 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=129, tag_tid=26, entry_tid=16897], [object_number=128, tag_tid=26, entry_tid=16896], [object_number=131, tag_tid=26, entry_tid=16895], [object_number=130, tag_tid=26, entry_tid=16894]]], state=connected] 2024-09-12T06:36:29.871 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:29.872 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=129, tag_tid=27, entry_tid=1' 2024-09-12T06:36:29.873 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-12T06:36:29.874 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:36:29.874 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:36:29.874 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:36:29.874 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-12T06:36:29.874 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:36:29.874 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:36:29.874 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:36:29.874 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:36:29.874 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-12T06:36:29.874 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-test.status 2024-09-12T06:36:29.874 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:36:29.875 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:29.875 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-12T06:36:29.875 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:29.922 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 32 2024-09-12T06:36:29.922 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 32 2024-09-12T06:36:29.922 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:36:29.922 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=129, tag_tid=27, entry_tid=1], [object_number=128, tag_tid=27, entry_tid=0], [object_number=131, tag_tid=26, entry_tid=19727], [object_number=130, tag_tid=26, entry_tid=19726]]], state=connected] 2024-09-12T06:36:29.922 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=129, tag_tid=26, entry_tid=16897], [object_number=128, tag_tid=26, entry_tid=16896], [object_number=131, tag_tid=26, entry_tid=16895], [object_number=130, tag_tid=26, entry_tid=16894]]], state=connected] 2024-09-12T06:36:29.925 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:29.927 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=129, tag_tid=26, entry_tid=16897' 2024-09-12T06:36:29.927 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=129, tag_tid=27, entry_tid=1' -a 'object_number=129, tag_tid=27, entry_tid=1' = 'object_number=129, tag_tid=26, entry_tid=16897' 2024-09-12T06:36:29.927 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 'object_number=129, tag_tid=26, entry_tid=16897' '!=' 'object_number=131, tag_tid=26, entry_tid=16703' 2024-09-12T06:36:29.927 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ break 2024-09-12T06:36:29.928 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 'object_number=129, tag_tid=26, entry_tid=16897' = 'object_number=131, tag_tid=26, entry_tid=16703' 2024-09-12T06:36:29.928 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ last_mirror_pos='object_number=129, tag_tid=26, entry_tid=16897' 2024-09-12T06:36:29.928 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=129, tag_tid=27, entry_tid=1' 2024-09-12T06:36:29.928 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-12T06:36:29.928 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:36:29.928 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_tag=27 2024-09-12T06:36:29.929 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=129, tag_tid=26, entry_tid=16897' 2024-09-12T06:36:29.929 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-12T06:36:29.929 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:36:29.930 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_tag=26 2024-09-12T06:36:29.930 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=129, tag_tid=27, entry_tid=1' 2024-09-12T06:36:29.930 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-12T06:36:29.931 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:36:29.931 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_entry=1 2024-09-12T06:36:29.932 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=129, tag_tid=26, entry_tid=16897' 2024-09-12T06:36:29.932 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-12T06:36:29.932 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:36:29.933 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_entry=16897 2024-09-12T06:36:29.933 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 27 = 26 -a 1 -le 16897 2024-09-12T06:36:29.933 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:36:29.934 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:36:29.934 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:36:30.135 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror test 2024-09-12T06:36:30.136 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:36:30.136 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:36:30.136 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:36:30.136 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:36:30.136 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n test ']' 2024-09-12T06:36:30.136 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-12T06:36:30.136 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-12T06:36:30.136 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:36:30.136 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:36:30.136 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:36:30.136 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:36:30.137 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:36:30.137 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:36:30.137 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:36:30.137 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:36:30.137 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:36:30.139 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:36:30.140 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-12T06:36:30.140 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:36:30.140 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:36:30.140 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:36:30.140 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:36:30.140 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:36:30.140 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:36:30.140 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:36:30.140 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:36:30.140 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:36:30.140 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:36:30.140 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:36:30.141 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:36:30.141 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:36:30.141 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:36:30.141 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:36:30.141 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:36:30.141 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:36:30.141 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:36:30.141 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:36:30.141 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:36:30.141 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:36:30.141 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:36:30.141 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:36:30.141 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:36:30.142 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:36:30.142 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:36:30.142 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:36:30.142 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:36:30.142 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:36:30.142 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:36:30.157 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:36:30.157 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:36:30.157 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:36:30.157 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-12T06:36:30.247 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:36:30.247 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-12T06:36:30.247 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:36:30.247 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:36:30.247 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:36:30.247 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-12T06:36:30.247 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:36:30.247 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:36:30.247 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:36:30.248 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:36:30.248 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-12T06:36:30.248 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-test.status 2024-09-12T06:36:30.248 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:36:30.248 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:30.248 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-12T06:36:30.249 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:30.296 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 32 2024-09-12T06:36:30.297 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 32 2024-09-12T06:36:30.297 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:36:30.297 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=129, tag_tid=27, entry_tid=1], [object_number=128, tag_tid=27, entry_tid=0], [object_number=131, tag_tid=26, entry_tid=19727], [object_number=130, tag_tid=26, entry_tid=19726]]], state=connected] 2024-09-12T06:36:30.297 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=128, tag_tid=26, entry_tid=17072], [object_number=131, tag_tid=26, entry_tid=17071], [object_number=130, tag_tid=26, entry_tid=17070], [object_number=129, tag_tid=26, entry_tid=17069]]], state=connected] 2024-09-12T06:36:30.299 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:30.301 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=129, tag_tid=27, entry_tid=1' 2024-09-12T06:36:30.302 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-12T06:36:30.302 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:36:30.302 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:36:30.302 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:36:30.302 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-12T06:36:30.302 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:36:30.302 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:36:30.303 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:36:30.303 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:36:30.303 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-12T06:36:30.303 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-test.status 2024-09-12T06:36:30.303 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:36:30.303 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:30.303 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-12T06:36:30.304 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:30.351 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 32 2024-09-12T06:36:30.351 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 32 2024-09-12T06:36:30.352 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:36:30.352 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=129, tag_tid=27, entry_tid=1], [object_number=128, tag_tid=27, entry_tid=0], [object_number=131, tag_tid=26, entry_tid=19727], [object_number=130, tag_tid=26, entry_tid=19726]]], state=connected] 2024-09-12T06:36:30.352 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=128, tag_tid=26, entry_tid=17072], [object_number=131, tag_tid=26, entry_tid=17071], [object_number=130, tag_tid=26, entry_tid=17070], [object_number=129, tag_tid=26, entry_tid=17069]]], state=connected] 2024-09-12T06:36:30.354 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:30.356 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=128, tag_tid=26, entry_tid=17072' 2024-09-12T06:36:30.356 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=129, tag_tid=27, entry_tid=1' -a 'object_number=129, tag_tid=27, entry_tid=1' = 'object_number=128, tag_tid=26, entry_tid=17072' 2024-09-12T06:36:30.357 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 'object_number=128, tag_tid=26, entry_tid=17072' '!=' 'object_number=129, tag_tid=26, entry_tid=16897' 2024-09-12T06:36:30.357 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ break 2024-09-12T06:36:30.357 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 'object_number=128, tag_tid=26, entry_tid=17072' = 'object_number=129, tag_tid=26, entry_tid=16897' 2024-09-12T06:36:30.357 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ last_mirror_pos='object_number=128, tag_tid=26, entry_tid=17072' 2024-09-12T06:36:30.357 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=129, tag_tid=27, entry_tid=1' 2024-09-12T06:36:30.357 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-12T06:36:30.357 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:36:30.358 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_tag=27 2024-09-12T06:36:30.358 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=128, tag_tid=26, entry_tid=17072' 2024-09-12T06:36:30.358 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-12T06:36:30.358 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:36:30.359 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_tag=26 2024-09-12T06:36:30.359 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=129, tag_tid=27, entry_tid=1' 2024-09-12T06:36:30.359 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-12T06:36:30.359 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:36:30.360 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_entry=1 2024-09-12T06:36:30.361 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=128, tag_tid=26, entry_tid=17072' 2024-09-12T06:36:30.361 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-12T06:36:30.361 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:36:30.362 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_entry=17072 2024-09-12T06:36:30.362 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 27 = 26 -a 1 -le 17072 2024-09-12T06:36:30.362 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:36:30.363 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:36:30.363 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:36:30.564 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror test 2024-09-12T06:36:30.565 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:36:30.565 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:36:30.565 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:36:30.565 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:36:30.565 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n test ']' 2024-09-12T06:36:30.565 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-12T06:36:30.565 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-12T06:36:30.565 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:36:30.565 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:36:30.565 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:36:30.565 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:36:30.565 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:36:30.566 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:36:30.566 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:36:30.566 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:36:30.566 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:36:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:36:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-12T06:36:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:36:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:36:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:36:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:36:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:36:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:36:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:36:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:36:30.570 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:36:30.570 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:36:30.570 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:36:30.570 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:36:30.570 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:36:30.570 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:36:30.570 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:36:30.570 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:36:30.570 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:36:30.570 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:36:30.570 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:36:30.570 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:36:30.570 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:36:30.571 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:36:30.571 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:36:30.571 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:36:30.571 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:36:30.571 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:36:30.571 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:36:30.571 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:36:30.571 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:36:30.571 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:36:30.586 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:36:30.586 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:36:30.586 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:36:30.586 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-12T06:36:30.674 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:36:30.674 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-12T06:36:30.674 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:36:30.674 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:36:30.674 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:36:30.675 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-12T06:36:30.675 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:36:30.675 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:36:30.675 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:36:30.675 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:36:30.675 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-12T06:36:30.675 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-test.status 2024-09-12T06:36:30.675 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:36:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-12T06:36:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:30.723 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 32 2024-09-12T06:36:30.723 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 32 2024-09-12T06:36:30.723 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:36:30.723 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=129, tag_tid=27, entry_tid=1], [object_number=128, tag_tid=27, entry_tid=0], [object_number=131, tag_tid=26, entry_tid=19727], [object_number=130, tag_tid=26, entry_tid=19726]]], state=connected] 2024-09-12T06:36:30.723 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=131, tag_tid=26, entry_tid=17243], [object_number=130, tag_tid=26, entry_tid=17242], [object_number=129, tag_tid=26, entry_tid=17241], [object_number=128, tag_tid=26, entry_tid=17240]]], state=connected] 2024-09-12T06:36:30.726 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:30.727 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=129, tag_tid=27, entry_tid=1' 2024-09-12T06:36:30.728 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-12T06:36:30.728 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:36:30.728 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:36:30.729 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:36:30.729 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-12T06:36:30.729 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:36:30.729 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:36:30.729 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:36:30.729 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:36:30.729 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-12T06:36:30.729 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-test.status 2024-09-12T06:36:30.729 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:36:30.730 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:30.730 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-12T06:36:30.730 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:30.777 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 32 2024-09-12T06:36:30.778 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 32 2024-09-12T06:36:30.778 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:36:30.778 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=129, tag_tid=27, entry_tid=1], [object_number=128, tag_tid=27, entry_tid=0], [object_number=131, tag_tid=26, entry_tid=19727], [object_number=130, tag_tid=26, entry_tid=19726]]], state=connected] 2024-09-12T06:36:30.778 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=131, tag_tid=26, entry_tid=17243], [object_number=130, tag_tid=26, entry_tid=17242], [object_number=129, tag_tid=26, entry_tid=17241], [object_number=128, tag_tid=26, entry_tid=17240]]], state=connected] 2024-09-12T06:36:30.780 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:30.782 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=131, tag_tid=26, entry_tid=17243' 2024-09-12T06:36:30.783 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=129, tag_tid=27, entry_tid=1' -a 'object_number=129, tag_tid=27, entry_tid=1' = 'object_number=131, tag_tid=26, entry_tid=17243' 2024-09-12T06:36:30.783 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 'object_number=131, tag_tid=26, entry_tid=17243' '!=' 'object_number=128, tag_tid=26, entry_tid=17072' 2024-09-12T06:36:30.783 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ break 2024-09-12T06:36:30.783 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 'object_number=131, tag_tid=26, entry_tid=17243' = 'object_number=128, tag_tid=26, entry_tid=17072' 2024-09-12T06:36:30.783 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ last_mirror_pos='object_number=131, tag_tid=26, entry_tid=17243' 2024-09-12T06:36:30.783 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=129, tag_tid=27, entry_tid=1' 2024-09-12T06:36:30.783 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-12T06:36:30.783 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:36:30.784 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_tag=27 2024-09-12T06:36:30.784 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=131, tag_tid=26, entry_tid=17243' 2024-09-12T06:36:30.784 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-12T06:36:30.784 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:36:30.785 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_tag=26 2024-09-12T06:36:30.786 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=129, tag_tid=27, entry_tid=1' 2024-09-12T06:36:30.786 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-12T06:36:30.786 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:36:30.787 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_entry=1 2024-09-12T06:36:30.787 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=131, tag_tid=26, entry_tid=17243' 2024-09-12T06:36:30.787 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-12T06:36:30.787 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:36:30.789 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_entry=17243 2024-09-12T06:36:30.789 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 27 = 26 -a 1 -le 17243 2024-09-12T06:36:30.789 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:36:30.789 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:36:30.789 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:36:30.990 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror test 2024-09-12T06:36:30.991 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:36:30.991 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:36:30.991 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:36:30.991 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:36:30.991 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n test ']' 2024-09-12T06:36:30.991 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-12T06:36:30.991 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-12T06:36:30.991 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:36:30.991 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:36:30.991 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:36:30.991 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:36:30.991 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:36:30.992 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:36:30.992 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:36:30.992 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:36:30.992 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:36:30.995 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:36:30.995 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-12T06:36:30.995 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:36:30.995 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:36:30.995 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:36:30.995 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:36:30.995 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:36:30.995 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:36:30.996 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:36:30.996 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:36:30.996 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:36:30.996 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:36:30.996 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:36:30.996 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:36:30.996 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:36:30.996 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:36:30.996 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:36:30.996 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:36:30.996 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:36:30.996 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:36:30.996 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:36:30.997 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:36:30.997 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:36:30.997 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:36:30.997 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:36:30.997 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:36:30.997 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:36:30.997 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:36:30.997 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:36:30.997 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:36:30.997 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:36:30.997 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:36:31.013 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:36:31.013 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:36:31.013 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:36:31.014 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-12T06:36:31.101 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:36:31.102 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-12T06:36:31.102 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:36:31.102 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:36:31.102 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:36:31.102 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-12T06:36:31.102 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:36:31.102 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:36:31.102 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:36:31.102 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:36:31.102 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-12T06:36:31.102 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-test.status 2024-09-12T06:36:31.102 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:36:31.103 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:31.103 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-12T06:36:31.103 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:31.151 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 32 2024-09-12T06:36:31.151 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 32 2024-09-12T06:36:31.151 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:36:31.151 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=129, tag_tid=27, entry_tid=1], [object_number=128, tag_tid=27, entry_tid=0], [object_number=131, tag_tid=26, entry_tid=19727], [object_number=130, tag_tid=26, entry_tid=19726]]], state=connected] 2024-09-12T06:36:31.151 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=129, tag_tid=26, entry_tid=17421], [object_number=128, tag_tid=26, entry_tid=17420], [object_number=131, tag_tid=26, entry_tid=17419], [object_number=130, tag_tid=26, entry_tid=17418]]], state=connected] 2024-09-12T06:36:31.154 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:31.155 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=129, tag_tid=27, entry_tid=1' 2024-09-12T06:36:31.156 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-12T06:36:31.156 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:36:31.156 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:36:31.156 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:36:31.157 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-12T06:36:31.157 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:36:31.157 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:36:31.157 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:36:31.157 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:36:31.157 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-12T06:36:31.157 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-test.status 2024-09-12T06:36:31.157 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:36:31.158 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:31.158 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-12T06:36:31.158 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:31.205 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 32 2024-09-12T06:36:31.205 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 32 2024-09-12T06:36:31.205 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:36:31.205 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=129, tag_tid=27, entry_tid=1], [object_number=128, tag_tid=27, entry_tid=0], [object_number=131, tag_tid=26, entry_tid=19727], [object_number=130, tag_tid=26, entry_tid=19726]]], state=connected] 2024-09-12T06:36:31.205 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=129, tag_tid=26, entry_tid=17421], [object_number=128, tag_tid=26, entry_tid=17420], [object_number=131, tag_tid=26, entry_tid=17419], [object_number=130, tag_tid=26, entry_tid=17418]]], state=connected] 2024-09-12T06:36:31.208 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:31.210 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=129, tag_tid=26, entry_tid=17421' 2024-09-12T06:36:31.210 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=129, tag_tid=27, entry_tid=1' -a 'object_number=129, tag_tid=27, entry_tid=1' = 'object_number=129, tag_tid=26, entry_tid=17421' 2024-09-12T06:36:31.210 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 'object_number=129, tag_tid=26, entry_tid=17421' '!=' 'object_number=131, tag_tid=26, entry_tid=17243' 2024-09-12T06:36:31.210 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ break 2024-09-12T06:36:31.210 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 'object_number=129, tag_tid=26, entry_tid=17421' = 'object_number=131, tag_tid=26, entry_tid=17243' 2024-09-12T06:36:31.211 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ last_mirror_pos='object_number=129, tag_tid=26, entry_tid=17421' 2024-09-12T06:36:31.211 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=129, tag_tid=27, entry_tid=1' 2024-09-12T06:36:31.211 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-12T06:36:31.211 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:36:31.211 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_tag=27 2024-09-12T06:36:31.212 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=129, tag_tid=26, entry_tid=17421' 2024-09-12T06:36:31.212 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-12T06:36:31.212 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:36:31.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_tag=26 2024-09-12T06:36:31.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=129, tag_tid=27, entry_tid=1' 2024-09-12T06:36:31.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-12T06:36:31.214 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:36:31.214 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_entry=1 2024-09-12T06:36:31.215 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=129, tag_tid=26, entry_tid=17421' 2024-09-12T06:36:31.215 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-12T06:36:31.215 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:36:31.216 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_entry=17421 2024-09-12T06:36:31.216 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 27 = 26 -a 1 -le 17421 2024-09-12T06:36:31.216 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:36:31.217 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:36:31.217 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:36:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror test 2024-09-12T06:36:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:36:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:36:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:36:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:36:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n test ']' 2024-09-12T06:36:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-12T06:36:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-12T06:36:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:36:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:36:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:36:31.420 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:36:31.420 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:36:31.420 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:36:31.420 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:36:31.420 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:36:31.420 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:36:31.423 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:36:31.423 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-12T06:36:31.423 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:36:31.423 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:36:31.423 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:36:31.423 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:36:31.423 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:36:31.424 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:36:31.424 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:36:31.424 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:36:31.424 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:36:31.424 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:36:31.424 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:36:31.424 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:36:31.424 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:36:31.424 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:36:31.424 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:36:31.424 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:36:31.424 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:36:31.424 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:36:31.425 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:36:31.425 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:36:31.425 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:36:31.425 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:36:31.425 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:36:31.425 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:36:31.425 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:36:31.425 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:36:31.425 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:36:31.425 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:36:31.425 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:36:31.425 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:36:31.440 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:36:31.440 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:36:31.440 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:36:31.440 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-12T06:36:31.527 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:36:31.528 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-12T06:36:31.528 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:36:31.528 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:36:31.528 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:36:31.528 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-12T06:36:31.528 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:36:31.528 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:36:31.528 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:36:31.528 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:36:31.528 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-12T06:36:31.528 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-test.status 2024-09-12T06:36:31.528 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:36:31.529 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:31.529 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-12T06:36:31.529 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:31.577 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 32 2024-09-12T06:36:31.577 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 32 2024-09-12T06:36:31.577 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:36:31.577 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=129, tag_tid=27, entry_tid=1], [object_number=128, tag_tid=27, entry_tid=0], [object_number=131, tag_tid=26, entry_tid=19727], [object_number=130, tag_tid=26, entry_tid=19726]]], state=connected] 2024-09-12T06:36:31.577 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=131, tag_tid=26, entry_tid=17595], [object_number=130, tag_tid=26, entry_tid=17594], [object_number=129, tag_tid=26, entry_tid=17593], [object_number=128, tag_tid=26, entry_tid=17592]]], state=connected] 2024-09-12T06:36:31.579 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:31.581 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=129, tag_tid=27, entry_tid=1' 2024-09-12T06:36:31.582 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-12T06:36:31.582 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:36:31.582 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:36:31.582 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:36:31.582 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-12T06:36:31.583 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:36:31.583 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:36:31.583 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:36:31.583 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:36:31.583 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-12T06:36:31.583 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-test.status 2024-09-12T06:36:31.583 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:36:31.583 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:31.584 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-12T06:36:31.584 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:31.632 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 32 2024-09-12T06:36:31.632 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 32 2024-09-12T06:36:31.632 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:36:31.632 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=129, tag_tid=27, entry_tid=1], [object_number=128, tag_tid=27, entry_tid=0], [object_number=131, tag_tid=26, entry_tid=19727], [object_number=130, tag_tid=26, entry_tid=19726]]], state=connected] 2024-09-12T06:36:31.632 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=131, tag_tid=26, entry_tid=17595], [object_number=130, tag_tid=26, entry_tid=17594], [object_number=129, tag_tid=26, entry_tid=17593], [object_number=128, tag_tid=26, entry_tid=17592]]], state=connected] 2024-09-12T06:36:31.635 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:31.637 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=131, tag_tid=26, entry_tid=17595' 2024-09-12T06:36:31.637 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=129, tag_tid=27, entry_tid=1' -a 'object_number=129, tag_tid=27, entry_tid=1' = 'object_number=131, tag_tid=26, entry_tid=17595' 2024-09-12T06:36:31.637 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 'object_number=131, tag_tid=26, entry_tid=17595' '!=' 'object_number=129, tag_tid=26, entry_tid=17421' 2024-09-12T06:36:31.637 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ break 2024-09-12T06:36:31.637 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 'object_number=131, tag_tid=26, entry_tid=17595' = 'object_number=129, tag_tid=26, entry_tid=17421' 2024-09-12T06:36:31.637 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ last_mirror_pos='object_number=131, tag_tid=26, entry_tid=17595' 2024-09-12T06:36:31.637 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=129, tag_tid=27, entry_tid=1' 2024-09-12T06:36:31.637 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-12T06:36:31.638 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:36:31.638 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_tag=27 2024-09-12T06:36:31.638 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=131, tag_tid=26, entry_tid=17595' 2024-09-12T06:36:31.639 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-12T06:36:31.639 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:36:31.639 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_tag=26 2024-09-12T06:36:31.640 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=129, tag_tid=27, entry_tid=1' 2024-09-12T06:36:31.640 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-12T06:36:31.640 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:36:31.641 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_entry=1 2024-09-12T06:36:31.642 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=131, tag_tid=26, entry_tid=17595' 2024-09-12T06:36:31.642 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-12T06:36:31.642 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:36:31.643 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_entry=17595 2024-09-12T06:36:31.643 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 27 = 26 -a 1 -le 17595 2024-09-12T06:36:31.643 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:36:31.643 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:36:31.643 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:36:31.845 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror test 2024-09-12T06:36:31.845 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:36:31.845 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:36:31.845 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:36:31.845 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:36:31.845 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n test ']' 2024-09-12T06:36:31.846 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-12T06:36:31.846 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-12T06:36:31.846 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:36:31.846 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:36:31.846 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:36:31.846 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:36:31.846 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:36:31.846 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:36:31.846 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:36:31.846 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:36:31.846 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:36:31.849 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:36:31.849 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-12T06:36:31.849 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:36:31.850 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:36:31.850 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:36:31.850 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:36:31.850 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:36:31.850 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:36:31.850 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:36:31.850 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:36:31.850 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:36:31.850 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:36:31.850 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:36:31.850 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:36:31.850 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:36:31.850 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:36:31.850 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:36:31.851 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:36:31.851 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:36:31.851 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:36:31.851 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:36:31.851 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:36:31.851 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:36:31.851 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:36:31.851 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:36:31.851 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:36:31.851 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:36:31.851 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:36:31.851 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:36:31.851 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:36:31.851 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:36:31.851 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:36:31.868 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:36:31.868 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:36:31.868 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:36:31.868 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-12T06:36:31.955 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:36:31.955 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-12T06:36:31.955 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:36:31.955 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:36:31.955 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:36:31.955 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-12T06:36:31.955 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:36:31.955 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:36:31.955 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:36:31.956 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:36:31.956 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-12T06:36:31.956 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-test.status 2024-09-12T06:36:31.956 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:36:31.956 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:31.956 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-12T06:36:31.957 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:32.004 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 32 2024-09-12T06:36:32.004 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 32 2024-09-12T06:36:32.004 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:36:32.004 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=129, tag_tid=27, entry_tid=1], [object_number=128, tag_tid=27, entry_tid=0], [object_number=131, tag_tid=26, entry_tid=19727], [object_number=130, tag_tid=26, entry_tid=19726]]], state=connected] 2024-09-12T06:36:32.004 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=129, tag_tid=26, entry_tid=17693], [object_number=128, tag_tid=26, entry_tid=17692], [object_number=131, tag_tid=26, entry_tid=17691], [object_number=130, tag_tid=26, entry_tid=17690]]], state=connected] 2024-09-12T06:36:32.007 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:32.009 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=129, tag_tid=27, entry_tid=1' 2024-09-12T06:36:32.010 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-12T06:36:32.010 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:36:32.010 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:36:32.010 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:36:32.010 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-12T06:36:32.010 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:36:32.010 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:36:32.010 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:36:32.010 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:36:32.010 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-12T06:36:32.010 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-test.status 2024-09-12T06:36:32.011 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:36:32.011 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:32.011 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-12T06:36:32.011 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:32.059 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 32 2024-09-12T06:36:32.059 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 32 2024-09-12T06:36:32.059 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:36:32.059 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=129, tag_tid=27, entry_tid=1], [object_number=128, tag_tid=27, entry_tid=0], [object_number=131, tag_tid=26, entry_tid=19727], [object_number=130, tag_tid=26, entry_tid=19726]]], state=connected] 2024-09-12T06:36:32.059 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=129, tag_tid=26, entry_tid=17693], [object_number=128, tag_tid=26, entry_tid=17692], [object_number=131, tag_tid=26, entry_tid=17691], [object_number=130, tag_tid=26, entry_tid=17690]]], state=connected] 2024-09-12T06:36:32.061 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:32.064 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=129, tag_tid=26, entry_tid=17693' 2024-09-12T06:36:32.064 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=129, tag_tid=27, entry_tid=1' -a 'object_number=129, tag_tid=27, entry_tid=1' = 'object_number=129, tag_tid=26, entry_tid=17693' 2024-09-12T06:36:32.064 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 'object_number=129, tag_tid=26, entry_tid=17693' '!=' 'object_number=131, tag_tid=26, entry_tid=17595' 2024-09-12T06:36:32.064 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ break 2024-09-12T06:36:32.064 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 'object_number=129, tag_tid=26, entry_tid=17693' = 'object_number=131, tag_tid=26, entry_tid=17595' 2024-09-12T06:36:32.064 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ last_mirror_pos='object_number=129, tag_tid=26, entry_tid=17693' 2024-09-12T06:36:32.064 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=129, tag_tid=27, entry_tid=1' 2024-09-12T06:36:32.064 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-12T06:36:32.064 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:36:32.065 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_tag=27 2024-09-12T06:36:32.065 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=129, tag_tid=26, entry_tid=17693' 2024-09-12T06:36:32.065 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-12T06:36:32.066 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:36:32.066 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_tag=26 2024-09-12T06:36:32.067 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=129, tag_tid=27, entry_tid=1' 2024-09-12T06:36:32.067 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-12T06:36:32.067 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:36:32.069 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_entry=1 2024-09-12T06:36:32.069 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=129, tag_tid=26, entry_tid=17693' 2024-09-12T06:36:32.069 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-12T06:36:32.069 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:36:32.070 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_entry=17693 2024-09-12T06:36:32.070 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 27 = 26 -a 1 -le 17693 2024-09-12T06:36:32.070 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:36:32.071 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:36:32.071 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:36:32.272 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror test 2024-09-12T06:36:32.272 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:36:32.273 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:36:32.273 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:36:32.273 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:36:32.273 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n test ']' 2024-09-12T06:36:32.273 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-12T06:36:32.273 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-12T06:36:32.273 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:36:32.273 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:36:32.273 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:36:32.273 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:36:32.273 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:36:32.273 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:36:32.273 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:36:32.274 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:36:32.274 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:36:32.277 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:36:32.277 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-12T06:36:32.277 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:36:32.277 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:36:32.277 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:36:32.277 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:36:32.277 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:36:32.277 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:36:32.277 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:36:32.277 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:36:32.278 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:36:32.278 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:36:32.278 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:36:32.278 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:36:32.278 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:36:32.278 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:36:32.278 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:36:32.278 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:36:32.278 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:36:32.278 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:36:32.278 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:36:32.278 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:36:32.279 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:36:32.279 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:36:32.279 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:36:32.279 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:36:32.279 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:36:32.279 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:36:32.279 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:36:32.279 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:36:32.279 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:36:32.279 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:36:32.293 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:36:32.294 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:36:32.294 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:36:32.294 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-12T06:36:32.382 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:36:32.382 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-12T06:36:32.382 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:36:32.382 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:36:32.382 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:36:32.382 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-12T06:36:32.382 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:36:32.382 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:36:32.383 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:36:32.383 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:36:32.383 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-12T06:36:32.383 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-test.status 2024-09-12T06:36:32.383 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:36:32.383 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:32.384 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-12T06:36:32.384 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:32.432 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 32 2024-09-12T06:36:32.432 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 32 2024-09-12T06:36:32.432 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:36:32.432 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=129, tag_tid=27, entry_tid=1], [object_number=128, tag_tid=27, entry_tid=0], [object_number=131, tag_tid=26, entry_tid=19727], [object_number=130, tag_tid=26, entry_tid=19726]]], state=connected] 2024-09-12T06:36:32.432 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=128, tag_tid=26, entry_tid=17880], [object_number=131, tag_tid=26, entry_tid=17879], [object_number=130, tag_tid=26, entry_tid=17878], [object_number=129, tag_tid=26, entry_tid=17877]]], state=connected] 2024-09-12T06:36:32.435 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:32.437 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=129, tag_tid=27, entry_tid=1' 2024-09-12T06:36:32.438 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-12T06:36:32.438 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:36:32.438 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:36:32.438 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:36:32.438 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-12T06:36:32.438 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:36:32.438 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:36:32.438 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:36:32.438 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:36:32.438 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-12T06:36:32.438 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-test.status 2024-09-12T06:36:32.438 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:36:32.439 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:32.439 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-12T06:36:32.439 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:32.487 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 32 2024-09-12T06:36:32.487 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 32 2024-09-12T06:36:32.487 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:36:32.487 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=129, tag_tid=27, entry_tid=1], [object_number=128, tag_tid=27, entry_tid=0], [object_number=131, tag_tid=26, entry_tid=19727], [object_number=130, tag_tid=26, entry_tid=19726]]], state=connected] 2024-09-12T06:36:32.488 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=128, tag_tid=26, entry_tid=17880], [object_number=131, tag_tid=26, entry_tid=17879], [object_number=130, tag_tid=26, entry_tid=17878], [object_number=129, tag_tid=26, entry_tid=17877]]], state=connected] 2024-09-12T06:36:32.490 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:32.492 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=128, tag_tid=26, entry_tid=17880' 2024-09-12T06:36:32.492 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=129, tag_tid=27, entry_tid=1' -a 'object_number=129, tag_tid=27, entry_tid=1' = 'object_number=128, tag_tid=26, entry_tid=17880' 2024-09-12T06:36:32.492 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 'object_number=128, tag_tid=26, entry_tid=17880' '!=' 'object_number=129, tag_tid=26, entry_tid=17693' 2024-09-12T06:36:32.492 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ break 2024-09-12T06:36:32.493 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 'object_number=128, tag_tid=26, entry_tid=17880' = 'object_number=129, tag_tid=26, entry_tid=17693' 2024-09-12T06:36:32.493 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ last_mirror_pos='object_number=128, tag_tid=26, entry_tid=17880' 2024-09-12T06:36:32.493 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=129, tag_tid=27, entry_tid=1' 2024-09-12T06:36:32.493 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-12T06:36:32.493 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:36:32.493 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_tag=27 2024-09-12T06:36:32.494 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=128, tag_tid=26, entry_tid=17880' 2024-09-12T06:36:32.494 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-12T06:36:32.494 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:36:32.495 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_tag=26 2024-09-12T06:36:32.495 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=129, tag_tid=27, entry_tid=1' 2024-09-12T06:36:32.495 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-12T06:36:32.495 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:36:32.496 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_entry=1 2024-09-12T06:36:32.497 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=128, tag_tid=26, entry_tid=17880' 2024-09-12T06:36:32.497 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-12T06:36:32.497 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:36:32.498 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_entry=17880 2024-09-12T06:36:32.498 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 27 = 26 -a 1 -le 17880 2024-09-12T06:36:32.498 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:36:32.498 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:36:32.499 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:36:32.700 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror test 2024-09-12T06:36:32.701 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:36:32.701 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:36:32.701 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:36:32.701 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:36:32.701 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n test ']' 2024-09-12T06:36:32.701 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-12T06:36:32.701 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-12T06:36:32.701 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:36:32.701 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:36:32.701 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:36:32.701 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:36:32.701 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:36:32.702 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:36:32.702 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:36:32.702 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:36:32.702 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:36:32.705 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:36:32.705 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-12T06:36:32.705 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:36:32.705 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:36:32.705 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:36:32.705 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:36:32.705 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:36:32.705 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:36:32.705 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:36:32.705 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:36:32.705 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:36:32.705 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:36:32.706 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:36:32.706 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:36:32.706 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:36:32.706 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:36:32.706 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:36:32.706 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:36:32.706 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:36:32.706 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:36:32.706 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:36:32.706 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:36:32.706 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:36:32.706 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:36:32.706 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:36:32.706 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:36:32.707 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:36:32.707 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:36:32.707 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:36:32.707 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:36:32.707 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:36:32.707 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:36:32.722 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:36:32.722 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:36:32.722 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:36:32.722 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-12T06:36:32.808 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:36:32.809 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-12T06:36:32.809 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:36:32.809 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:36:32.809 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:36:32.809 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-12T06:36:32.809 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:36:32.809 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:36:32.809 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:36:32.810 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:36:32.810 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-12T06:36:32.810 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-test.status 2024-09-12T06:36:32.810 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:36:32.810 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:32.810 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-12T06:36:32.811 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:32.858 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 32 2024-09-12T06:36:32.858 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 32 2024-09-12T06:36:32.858 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:36:32.858 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=129, tag_tid=27, entry_tid=1], [object_number=128, tag_tid=27, entry_tid=0], [object_number=131, tag_tid=26, entry_tid=19727], [object_number=130, tag_tid=26, entry_tid=19726]]], state=connected] 2024-09-12T06:36:32.858 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=128, tag_tid=26, entry_tid=18056], [object_number=131, tag_tid=26, entry_tid=18055], [object_number=130, tag_tid=26, entry_tid=18054], [object_number=129, tag_tid=26, entry_tid=18053]]], state=connected] 2024-09-12T06:36:32.861 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:32.863 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=129, tag_tid=27, entry_tid=1' 2024-09-12T06:36:32.863 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-12T06:36:32.864 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:36:32.864 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:36:32.864 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:36:32.864 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-12T06:36:32.864 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:36:32.864 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:36:32.864 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:36:32.864 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:36:32.864 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-12T06:36:32.864 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-test.status 2024-09-12T06:36:32.864 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:36:32.865 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:32.865 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-12T06:36:32.865 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:32.913 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 32 2024-09-12T06:36:32.913 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 32 2024-09-12T06:36:32.913 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:36:32.913 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=129, tag_tid=27, entry_tid=1], [object_number=128, tag_tid=27, entry_tid=0], [object_number=131, tag_tid=26, entry_tid=19727], [object_number=130, tag_tid=26, entry_tid=19726]]], state=connected] 2024-09-12T06:36:32.913 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=128, tag_tid=26, entry_tid=18056], [object_number=131, tag_tid=26, entry_tid=18055], [object_number=130, tag_tid=26, entry_tid=18054], [object_number=129, tag_tid=26, entry_tid=18053]]], state=connected] 2024-09-12T06:36:32.916 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:32.918 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=128, tag_tid=26, entry_tid=18056' 2024-09-12T06:36:32.918 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=129, tag_tid=27, entry_tid=1' -a 'object_number=129, tag_tid=27, entry_tid=1' = 'object_number=128, tag_tid=26, entry_tid=18056' 2024-09-12T06:36:32.918 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 'object_number=128, tag_tid=26, entry_tid=18056' '!=' 'object_number=128, tag_tid=26, entry_tid=17880' 2024-09-12T06:36:32.918 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ break 2024-09-12T06:36:32.918 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 'object_number=128, tag_tid=26, entry_tid=18056' = 'object_number=128, tag_tid=26, entry_tid=17880' 2024-09-12T06:36:32.918 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ last_mirror_pos='object_number=128, tag_tid=26, entry_tid=18056' 2024-09-12T06:36:32.918 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=129, tag_tid=27, entry_tid=1' 2024-09-12T06:36:32.919 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-12T06:36:32.919 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:36:32.919 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_tag=27 2024-09-12T06:36:32.920 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=128, tag_tid=26, entry_tid=18056' 2024-09-12T06:36:32.920 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-12T06:36:32.920 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:36:32.921 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_tag=26 2024-09-12T06:36:32.921 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=129, tag_tid=27, entry_tid=1' 2024-09-12T06:36:32.921 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-12T06:36:32.921 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:36:32.922 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_entry=1 2024-09-12T06:36:32.923 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=128, tag_tid=26, entry_tid=18056' 2024-09-12T06:36:32.923 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-12T06:36:32.923 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:36:32.924 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_entry=18056 2024-09-12T06:36:32.924 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 27 = 26 -a 1 -le 18056 2024-09-12T06:36:32.924 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:36:32.924 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:36:32.924 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:36:33.126 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror test 2024-09-12T06:36:33.126 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:36:33.126 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:36:33.127 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:36:33.127 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:36:33.127 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n test ']' 2024-09-12T06:36:33.127 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-12T06:36:33.127 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-12T06:36:33.127 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:36:33.127 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:36:33.127 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:36:33.127 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:36:33.127 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:36:33.127 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:36:33.127 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:36:33.127 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:36:33.127 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:36:33.130 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:36:33.130 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-12T06:36:33.131 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:36:33.131 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:36:33.131 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:36:33.131 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:36:33.131 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:36:33.131 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:36:33.131 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:36:33.131 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:36:33.131 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:36:33.131 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:36:33.131 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:36:33.131 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:36:33.131 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:36:33.131 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:36:33.131 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:36:33.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:36:33.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:36:33.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:36:33.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:36:33.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:36:33.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:36:33.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:36:33.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:36:33.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:36:33.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:36:33.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:36:33.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:36:33.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:36:33.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:36:33.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:36:33.147 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:36:33.148 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:36:33.148 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:36:33.148 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-12T06:36:33.235 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:36:33.236 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-12T06:36:33.236 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:36:33.236 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:36:33.236 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:36:33.237 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-12T06:36:33.237 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:36:33.237 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:36:33.237 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:36:33.237 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:36:33.237 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-12T06:36:33.237 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-test.status 2024-09-12T06:36:33.237 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:36:33.238 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:33.238 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-12T06:36:33.238 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:33.285 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 32 2024-09-12T06:36:33.286 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 32 2024-09-12T06:36:33.286 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:36:33.286 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=129, tag_tid=27, entry_tid=1], [object_number=128, tag_tid=27, entry_tid=0], [object_number=131, tag_tid=26, entry_tid=19727], [object_number=130, tag_tid=26, entry_tid=19726]]], state=connected] 2024-09-12T06:36:33.286 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=130, tag_tid=26, entry_tid=18234], [object_number=129, tag_tid=26, entry_tid=18233], [object_number=128, tag_tid=26, entry_tid=18232], [object_number=131, tag_tid=26, entry_tid=18231]]], state=connected] 2024-09-12T06:36:33.289 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:33.290 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=129, tag_tid=27, entry_tid=1' 2024-09-12T06:36:33.293 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-12T06:36:33.293 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:36:33.293 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:36:33.293 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:36:33.293 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-12T06:36:33.293 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:36:33.293 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:36:33.293 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:36:33.293 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:36:33.293 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-12T06:36:33.293 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-test.status 2024-09-12T06:36:33.293 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:36:33.293 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:33.294 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-12T06:36:33.294 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:33.342 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 32 2024-09-12T06:36:33.343 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 32 2024-09-12T06:36:33.343 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:36:33.343 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=129, tag_tid=27, entry_tid=1], [object_number=128, tag_tid=27, entry_tid=0], [object_number=131, tag_tid=26, entry_tid=19727], [object_number=130, tag_tid=26, entry_tid=19726]]], state=connected] 2024-09-12T06:36:33.343 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=130, tag_tid=26, entry_tid=18234], [object_number=129, tag_tid=26, entry_tid=18233], [object_number=128, tag_tid=26, entry_tid=18232], [object_number=131, tag_tid=26, entry_tid=18231]]], state=connected] 2024-09-12T06:36:33.345 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:33.348 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=130, tag_tid=26, entry_tid=18234' 2024-09-12T06:36:33.348 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=129, tag_tid=27, entry_tid=1' -a 'object_number=129, tag_tid=27, entry_tid=1' = 'object_number=130, tag_tid=26, entry_tid=18234' 2024-09-12T06:36:33.348 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 'object_number=130, tag_tid=26, entry_tid=18234' '!=' 'object_number=128, tag_tid=26, entry_tid=18056' 2024-09-12T06:36:33.348 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ break 2024-09-12T06:36:33.348 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 'object_number=130, tag_tid=26, entry_tid=18234' = 'object_number=128, tag_tid=26, entry_tid=18056' 2024-09-12T06:36:33.348 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ last_mirror_pos='object_number=130, tag_tid=26, entry_tid=18234' 2024-09-12T06:36:33.348 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=129, tag_tid=27, entry_tid=1' 2024-09-12T06:36:33.348 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-12T06:36:33.348 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:36:33.349 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_tag=27 2024-09-12T06:36:33.349 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=130, tag_tid=26, entry_tid=18234' 2024-09-12T06:36:33.349 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-12T06:36:33.349 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:36:33.350 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_tag=26 2024-09-12T06:36:33.351 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=129, tag_tid=27, entry_tid=1' 2024-09-12T06:36:33.351 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-12T06:36:33.351 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:36:33.352 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_entry=1 2024-09-12T06:36:33.352 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=130, tag_tid=26, entry_tid=18234' 2024-09-12T06:36:33.352 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-12T06:36:33.353 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:36:33.354 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_entry=18234 2024-09-12T06:36:33.354 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 27 = 26 -a 1 -le 18234 2024-09-12T06:36:33.354 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:36:33.354 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:36:33.354 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:36:33.556 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror test 2024-09-12T06:36:33.556 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:36:33.556 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:36:33.556 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:36:33.556 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:36:33.556 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n test ']' 2024-09-12T06:36:33.556 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-12T06:36:33.557 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-12T06:36:33.557 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:36:33.557 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:36:33.557 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:36:33.557 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:36:33.557 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:36:33.557 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:36:33.557 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:36:33.557 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:36:33.557 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:36:33.560 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:36:33.560 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-12T06:36:33.560 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:36:33.560 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:36:33.560 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:36:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:36:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:36:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:36:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:36:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:36:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:36:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:36:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:36:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:36:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:36:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:36:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:36:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:36:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:36:33.562 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:36:33.562 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:36:33.562 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:36:33.562 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:36:33.562 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:36:33.562 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:36:33.562 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:36:33.562 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:36:33.562 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:36:33.562 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:36:33.562 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:36:33.562 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:36:33.562 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:36:33.577 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:36:33.577 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:36:33.577 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:36:33.577 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-12T06:36:33.664 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:36:33.664 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-12T06:36:33.664 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:36:33.664 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:36:33.664 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:36:33.664 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-12T06:36:33.664 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:36:33.664 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:36:33.664 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:36:33.664 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:36:33.665 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-12T06:36:33.665 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-test.status 2024-09-12T06:36:33.665 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:36:33.665 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:33.665 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-12T06:36:33.665 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:33.713 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 32 2024-09-12T06:36:33.713 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 32 2024-09-12T06:36:33.713 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:36:33.713 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=129, tag_tid=27, entry_tid=1], [object_number=128, tag_tid=27, entry_tid=0], [object_number=131, tag_tid=26, entry_tid=19727], [object_number=130, tag_tid=26, entry_tid=19726]]], state=connected] 2024-09-12T06:36:33.713 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=131, tag_tid=26, entry_tid=18411], [object_number=130, tag_tid=26, entry_tid=18410], [object_number=129, tag_tid=26, entry_tid=18409], [object_number=128, tag_tid=26, entry_tid=18408]]], state=connected] 2024-09-12T06:36:33.716 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:33.717 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=129, tag_tid=27, entry_tid=1' 2024-09-12T06:36:33.719 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-12T06:36:33.719 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:36:33.719 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:36:33.719 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:36:33.719 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-12T06:36:33.719 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:36:33.719 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:36:33.719 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:36:33.719 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:36:33.719 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-12T06:36:33.720 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-test.status 2024-09-12T06:36:33.720 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:36:33.720 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:33.720 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-12T06:36:33.720 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:33.768 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 32 2024-09-12T06:36:33.768 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 32 2024-09-12T06:36:33.768 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:36:33.768 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=129, tag_tid=27, entry_tid=1], [object_number=128, tag_tid=27, entry_tid=0], [object_number=131, tag_tid=26, entry_tid=19727], [object_number=130, tag_tid=26, entry_tid=19726]]], state=connected] 2024-09-12T06:36:33.768 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=131, tag_tid=26, entry_tid=18411], [object_number=130, tag_tid=26, entry_tid=18410], [object_number=129, tag_tid=26, entry_tid=18409], [object_number=128, tag_tid=26, entry_tid=18408]]], state=connected] 2024-09-12T06:36:33.771 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:33.773 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=131, tag_tid=26, entry_tid=18411' 2024-09-12T06:36:33.773 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=129, tag_tid=27, entry_tid=1' -a 'object_number=129, tag_tid=27, entry_tid=1' = 'object_number=131, tag_tid=26, entry_tid=18411' 2024-09-12T06:36:33.773 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 'object_number=131, tag_tid=26, entry_tid=18411' '!=' 'object_number=130, tag_tid=26, entry_tid=18234' 2024-09-12T06:36:33.773 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ break 2024-09-12T06:36:33.773 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 'object_number=131, tag_tid=26, entry_tid=18411' = 'object_number=130, tag_tid=26, entry_tid=18234' 2024-09-12T06:36:33.773 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ last_mirror_pos='object_number=131, tag_tid=26, entry_tid=18411' 2024-09-12T06:36:33.773 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=129, tag_tid=27, entry_tid=1' 2024-09-12T06:36:33.774 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-12T06:36:33.774 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:36:33.774 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_tag=27 2024-09-12T06:36:33.775 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=131, tag_tid=26, entry_tid=18411' 2024-09-12T06:36:33.775 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-12T06:36:33.775 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:36:33.775 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_tag=26 2024-09-12T06:36:33.776 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=129, tag_tid=27, entry_tid=1' 2024-09-12T06:36:33.776 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-12T06:36:33.776 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:36:33.777 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_entry=1 2024-09-12T06:36:33.778 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=131, tag_tid=26, entry_tid=18411' 2024-09-12T06:36:33.778 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-12T06:36:33.778 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:36:33.779 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_entry=18411 2024-09-12T06:36:33.779 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 27 = 26 -a 1 -le 18411 2024-09-12T06:36:33.779 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:36:33.779 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:36:33.779 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:36:33.985 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror test 2024-09-12T06:36:33.985 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:36:33.985 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:36:33.985 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:36:33.985 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:36:33.986 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n test ']' 2024-09-12T06:36:33.986 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-12T06:36:33.986 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-12T06:36:33.986 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:36:33.986 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:36:33.986 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:36:33.986 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:36:33.986 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:36:33.986 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:36:33.986 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:36:33.986 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:36:33.986 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:36:33.986 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:36:33.987 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-12T06:36:33.987 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:36:33.987 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:36:33.987 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:36:33.987 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:36:33.987 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:36:33.987 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:36:33.987 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:36:33.987 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:36:33.987 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:36:33.987 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:36:33.987 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:36:33.987 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:36:33.987 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:36:33.988 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:36:33.988 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:36:33.988 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:36:33.988 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:36:33.988 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:36:33.988 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:36:33.988 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:36:33.988 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:36:33.988 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:36:33.988 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:36:33.988 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:36:33.988 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:36:33.988 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:36:33.988 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:36:33.989 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:36:33.989 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:36:33.989 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:36:34.003 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:36:34.003 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:36:34.003 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:36:34.003 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-12T06:36:34.089 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:36:34.090 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-12T06:36:34.090 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:36:34.090 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:36:34.091 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:36:34.091 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-12T06:36:34.091 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:36:34.091 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:36:34.091 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:36:34.091 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:36:34.091 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-12T06:36:34.091 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-test.status 2024-09-12T06:36:34.091 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:36:34.092 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:34.092 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-12T06:36:34.092 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:34.342 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 32 2024-09-12T06:36:34.342 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 32 2024-09-12T06:36:34.342 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:36:34.342 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=129, tag_tid=27, entry_tid=1], [object_number=128, tag_tid=27, entry_tid=0], [object_number=131, tag_tid=26, entry_tid=19727], [object_number=130, tag_tid=26, entry_tid=19726]]], state=connected] 2024-09-12T06:36:34.342 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=128, tag_tid=26, entry_tid=18588], [object_number=131, tag_tid=26, entry_tid=18587], [object_number=130, tag_tid=26, entry_tid=18586], [object_number=129, tag_tid=26, entry_tid=18585]]], state=connected] 2024-09-12T06:36:34.344 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:34.347 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=129, tag_tid=27, entry_tid=1' 2024-09-12T06:36:34.347 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-12T06:36:34.347 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:36:34.347 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:36:34.347 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:36:34.347 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-12T06:36:34.348 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:36:34.348 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:36:34.348 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:36:34.348 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:36:34.348 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-12T06:36:34.348 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-test.status 2024-09-12T06:36:34.348 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:36:34.349 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:34.349 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-12T06:36:34.349 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:34.400 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 32 2024-09-12T06:36:34.400 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 32 2024-09-12T06:36:34.400 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:36:34.400 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=129, tag_tid=27, entry_tid=1], [object_number=128, tag_tid=27, entry_tid=0], [object_number=131, tag_tid=26, entry_tid=19727], [object_number=130, tag_tid=26, entry_tid=19726]]], state=connected] 2024-09-12T06:36:34.401 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=128, tag_tid=26, entry_tid=18588], [object_number=131, tag_tid=26, entry_tid=18587], [object_number=130, tag_tid=26, entry_tid=18586], [object_number=129, tag_tid=26, entry_tid=18585]]], state=connected] 2024-09-12T06:36:34.403 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:34.405 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=128, tag_tid=26, entry_tid=18588' 2024-09-12T06:36:34.406 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=129, tag_tid=27, entry_tid=1' -a 'object_number=129, tag_tid=27, entry_tid=1' = 'object_number=128, tag_tid=26, entry_tid=18588' 2024-09-12T06:36:34.406 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 'object_number=128, tag_tid=26, entry_tid=18588' '!=' 'object_number=131, tag_tid=26, entry_tid=18411' 2024-09-12T06:36:34.406 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ break 2024-09-12T06:36:34.406 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 'object_number=128, tag_tid=26, entry_tid=18588' = 'object_number=131, tag_tid=26, entry_tid=18411' 2024-09-12T06:36:34.406 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ last_mirror_pos='object_number=128, tag_tid=26, entry_tid=18588' 2024-09-12T06:36:34.406 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=129, tag_tid=27, entry_tid=1' 2024-09-12T06:36:34.406 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-12T06:36:34.406 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:36:34.407 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_tag=27 2024-09-12T06:36:34.407 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=128, tag_tid=26, entry_tid=18588' 2024-09-12T06:36:34.407 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-12T06:36:34.407 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:36:34.408 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_tag=26 2024-09-12T06:36:34.409 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=129, tag_tid=27, entry_tid=1' 2024-09-12T06:36:34.409 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-12T06:36:34.409 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:36:34.409 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_entry=1 2024-09-12T06:36:34.410 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=128, tag_tid=26, entry_tid=18588' 2024-09-12T06:36:34.410 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-12T06:36:34.410 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:36:34.411 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_entry=18588 2024-09-12T06:36:34.412 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 27 = 26 -a 1 -le 18588 2024-09-12T06:36:34.412 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:36:34.412 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:36:34.412 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:36:34.614 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror test 2024-09-12T06:36:34.614 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:36:34.614 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:36:34.614 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:36:34.614 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:36:34.614 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n test ']' 2024-09-12T06:36:34.614 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-12T06:36:34.614 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-12T06:36:34.614 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:36:34.615 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:36:34.615 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:36:34.615 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:36:34.615 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:36:34.615 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:36:34.615 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:36:34.615 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:36:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:36:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:36:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-12T06:36:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:36:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:36:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:36:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:36:34.619 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:36:34.619 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:36:34.619 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:36:34.619 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:36:34.619 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:36:34.619 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:36:34.619 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:36:34.619 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:36:34.619 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:36:34.619 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:36:34.619 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:36:34.619 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:36:34.619 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:36:34.619 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:36:34.620 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:36:34.620 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:36:34.620 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:36:34.620 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:36:34.620 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:36:34.620 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:36:34.620 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:36:34.620 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:36:34.620 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:36:34.620 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:36:34.620 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:36:34.620 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:36:34.636 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:36:34.636 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:36:34.636 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:36:34.636 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-12T06:36:34.726 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:36:34.726 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-12T06:36:34.726 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:36:34.726 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:36:34.726 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:36:34.726 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-12T06:36:34.726 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:36:34.727 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:36:34.727 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:36:34.727 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:36:34.727 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-12T06:36:34.727 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-test.status 2024-09-12T06:36:34.727 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:36:34.727 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:34.727 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-12T06:36:34.727 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:34.775 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 32 2024-09-12T06:36:34.776 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 32 2024-09-12T06:36:34.776 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:36:34.776 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=129, tag_tid=27, entry_tid=1], [object_number=128, tag_tid=27, entry_tid=0], [object_number=131, tag_tid=26, entry_tid=19727], [object_number=130, tag_tid=26, entry_tid=19726]]], state=connected] 2024-09-12T06:36:34.776 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=128, tag_tid=26, entry_tid=18888], [object_number=131, tag_tid=26, entry_tid=18887], [object_number=130, tag_tid=26, entry_tid=18886], [object_number=129, tag_tid=26, entry_tid=18885]]], state=connected] 2024-09-12T06:36:34.778 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:34.780 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=129, tag_tid=27, entry_tid=1' 2024-09-12T06:36:34.781 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-12T06:36:34.781 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:36:34.781 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:36:34.781 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:36:34.782 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-12T06:36:34.782 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:36:34.782 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:36:34.782 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:36:34.782 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:36:34.782 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-12T06:36:34.782 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-test.status 2024-09-12T06:36:34.782 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:36:34.783 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:34.783 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-12T06:36:34.783 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:34.833 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 32 2024-09-12T06:36:34.833 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 32 2024-09-12T06:36:34.833 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:36:34.833 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=129, tag_tid=27, entry_tid=1], [object_number=128, tag_tid=27, entry_tid=0], [object_number=131, tag_tid=26, entry_tid=19727], [object_number=130, tag_tid=26, entry_tid=19726]]], state=connected] 2024-09-12T06:36:34.833 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=128, tag_tid=26, entry_tid=18888], [object_number=131, tag_tid=26, entry_tid=18887], [object_number=130, tag_tid=26, entry_tid=18886], [object_number=129, tag_tid=26, entry_tid=18885]]], state=connected] 2024-09-12T06:36:34.836 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:34.839 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=128, tag_tid=26, entry_tid=18888' 2024-09-12T06:36:34.839 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=129, tag_tid=27, entry_tid=1' -a 'object_number=129, tag_tid=27, entry_tid=1' = 'object_number=128, tag_tid=26, entry_tid=18888' 2024-09-12T06:36:34.839 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 'object_number=128, tag_tid=26, entry_tid=18888' '!=' 'object_number=128, tag_tid=26, entry_tid=18588' 2024-09-12T06:36:34.839 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ break 2024-09-12T06:36:34.839 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 'object_number=128, tag_tid=26, entry_tid=18888' = 'object_number=128, tag_tid=26, entry_tid=18588' 2024-09-12T06:36:34.839 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ last_mirror_pos='object_number=128, tag_tid=26, entry_tid=18888' 2024-09-12T06:36:34.839 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=129, tag_tid=27, entry_tid=1' 2024-09-12T06:36:34.839 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-12T06:36:34.839 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:36:34.840 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_tag=27 2024-09-12T06:36:34.840 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=128, tag_tid=26, entry_tid=18888' 2024-09-12T06:36:34.840 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-12T06:36:34.840 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:36:34.841 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_tag=26 2024-09-12T06:36:34.842 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=129, tag_tid=27, entry_tid=1' 2024-09-12T06:36:34.842 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-12T06:36:34.842 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:36:34.843 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_entry=1 2024-09-12T06:36:34.843 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=128, tag_tid=26, entry_tid=18888' 2024-09-12T06:36:34.843 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-12T06:36:34.844 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:36:34.845 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_entry=18888 2024-09-12T06:36:34.845 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 27 = 26 -a 1 -le 18888 2024-09-12T06:36:34.845 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:36:34.845 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:36:34.845 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:36:35.046 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror test 2024-09-12T06:36:35.047 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:36:35.047 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:36:35.047 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:36:35.047 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:36:35.047 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n test ']' 2024-09-12T06:36:35.047 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-12T06:36:35.047 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-12T06:36:35.047 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:36:35.047 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:36:35.047 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:36:35.047 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:36:35.047 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:36:35.047 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:36:35.047 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:36:35.047 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:36:35.050 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:36:35.050 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:36:35.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-12T06:36:35.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:36:35.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:36:35.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:36:35.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:36:35.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:36:35.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:36:35.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:36:35.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:36:35.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:36:35.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:36:35.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:36:35.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:36:35.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:36:35.052 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:36:35.052 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:36:35.052 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:36:35.052 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:36:35.052 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:36:35.052 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:36:35.052 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:36:35.052 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:36:35.052 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:36:35.052 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:36:35.052 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:36:35.052 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:36:35.052 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:36:35.052 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:36:35.053 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:36:35.053 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:36:35.053 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:36:35.068 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:36:35.068 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:36:35.068 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:36:35.068 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-12T06:36:35.157 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:36:35.158 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-12T06:36:35.158 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:36:35.158 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:36:35.158 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:36:35.158 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-12T06:36:35.158 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:36:35.158 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:36:35.158 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:36:35.158 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:36:35.158 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-12T06:36:35.158 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-test.status 2024-09-12T06:36:35.158 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:36:35.159 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:35.159 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-12T06:36:35.159 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:35.207 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 32 2024-09-12T06:36:35.207 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 32 2024-09-12T06:36:35.208 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:36:35.208 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=129, tag_tid=27, entry_tid=1], [object_number=128, tag_tid=27, entry_tid=0], [object_number=131, tag_tid=26, entry_tid=19727], [object_number=130, tag_tid=26, entry_tid=19726]]], state=connected] 2024-09-12T06:36:35.208 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=128, tag_tid=26, entry_tid=19080], [object_number=131, tag_tid=26, entry_tid=19079], [object_number=130, tag_tid=26, entry_tid=19078], [object_number=129, tag_tid=26, entry_tid=19077]]], state=connected] 2024-09-12T06:36:35.210 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:35.212 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=129, tag_tid=27, entry_tid=1' 2024-09-12T06:36:35.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-12T06:36:35.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:36:35.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:36:35.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:36:35.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-12T06:36:35.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:36:35.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:36:35.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:36:35.214 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:36:35.214 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-12T06:36:35.214 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-test.status 2024-09-12T06:36:35.214 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:36:35.214 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:35.214 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-12T06:36:35.215 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:35.262 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 32 2024-09-12T06:36:35.263 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 32 2024-09-12T06:36:35.263 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:36:35.263 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=129, tag_tid=27, entry_tid=1], [object_number=128, tag_tid=27, entry_tid=0], [object_number=131, tag_tid=26, entry_tid=19727], [object_number=130, tag_tid=26, entry_tid=19726]]], state=connected] 2024-09-12T06:36:35.263 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=128, tag_tid=26, entry_tid=19080], [object_number=131, tag_tid=26, entry_tid=19079], [object_number=130, tag_tid=26, entry_tid=19078], [object_number=129, tag_tid=26, entry_tid=19077]]], state=connected] 2024-09-12T06:36:35.265 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:35.268 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=128, tag_tid=26, entry_tid=19080' 2024-09-12T06:36:35.268 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=129, tag_tid=27, entry_tid=1' -a 'object_number=129, tag_tid=27, entry_tid=1' = 'object_number=128, tag_tid=26, entry_tid=19080' 2024-09-12T06:36:35.268 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 'object_number=128, tag_tid=26, entry_tid=19080' '!=' 'object_number=128, tag_tid=26, entry_tid=18888' 2024-09-12T06:36:35.268 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ break 2024-09-12T06:36:35.268 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 'object_number=128, tag_tid=26, entry_tid=19080' = 'object_number=128, tag_tid=26, entry_tid=18888' 2024-09-12T06:36:35.268 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ last_mirror_pos='object_number=128, tag_tid=26, entry_tid=19080' 2024-09-12T06:36:35.268 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=129, tag_tid=27, entry_tid=1' 2024-09-12T06:36:35.268 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-12T06:36:35.268 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:36:35.269 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_tag=27 2024-09-12T06:36:35.269 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=128, tag_tid=26, entry_tid=19080' 2024-09-12T06:36:35.269 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-12T06:36:35.269 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:36:35.270 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_tag=26 2024-09-12T06:36:35.271 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=129, tag_tid=27, entry_tid=1' 2024-09-12T06:36:35.271 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-12T06:36:35.271 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:36:35.272 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_entry=1 2024-09-12T06:36:35.273 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=128, tag_tid=26, entry_tid=19080' 2024-09-12T06:36:35.273 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-12T06:36:35.273 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:36:35.274 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_entry=19080 2024-09-12T06:36:35.274 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 27 = 26 -a 1 -le 19080 2024-09-12T06:36:35.274 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:36:35.274 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:36:35.274 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:36:35.476 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror test 2024-09-12T06:36:35.476 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:36:35.476 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:36:35.476 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:36:35.476 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:36:35.476 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n test ']' 2024-09-12T06:36:35.476 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-12T06:36:35.476 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-12T06:36:35.476 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:36:35.477 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:36:35.477 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:36:35.477 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:36:35.477 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:36:35.477 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:36:35.477 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:36:35.477 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:36:35.477 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:36:35.480 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:36:35.480 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-12T06:36:35.480 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:36:35.480 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:36:35.480 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:36:35.480 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:36:35.480 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:36:35.480 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:36:35.480 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:36:35.480 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:36:35.480 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:36:35.480 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:36:35.481 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:36:35.481 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:36:35.481 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:36:35.481 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:36:35.481 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:36:35.481 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:36:35.481 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:36:35.481 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:36:35.481 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:36:35.481 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:36:35.481 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:36:35.481 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:36:35.481 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:36:35.481 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:36:35.481 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:36:35.482 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:36:35.482 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:36:35.482 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:36:35.482 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:36:35.482 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:36:35.497 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:36:35.497 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:36:35.498 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:36:35.498 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-12T06:36:35.587 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:36:35.587 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-12T06:36:35.587 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:36:35.587 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:36:35.588 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:36:35.588 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-12T06:36:35.588 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:36:35.588 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:36:35.588 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:36:35.588 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:36:35.588 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-12T06:36:35.588 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-test.status 2024-09-12T06:36:35.588 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:36:35.589 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:35.589 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-12T06:36:35.589 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:35.636 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 32 2024-09-12T06:36:35.636 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 32 2024-09-12T06:36:35.636 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:36:35.636 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=129, tag_tid=27, entry_tid=1], [object_number=128, tag_tid=27, entry_tid=0], [object_number=131, tag_tid=26, entry_tid=19727], [object_number=130, tag_tid=26, entry_tid=19726]]], state=connected] 2024-09-12T06:36:35.637 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=130, tag_tid=26, entry_tid=19258], [object_number=129, tag_tid=26, entry_tid=19257], [object_number=128, tag_tid=26, entry_tid=19256], [object_number=131, tag_tid=26, entry_tid=19255]]], state=connected] 2024-09-12T06:36:35.639 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:35.642 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=129, tag_tid=27, entry_tid=1' 2024-09-12T06:36:35.642 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-12T06:36:35.642 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:36:35.642 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:36:35.642 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:36:35.642 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-12T06:36:35.642 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:36:35.642 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:36:35.642 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:36:35.642 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:36:35.642 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-12T06:36:35.642 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-test.status 2024-09-12T06:36:35.643 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:36:35.643 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:35.643 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-12T06:36:35.643 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:35.691 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 32 2024-09-12T06:36:35.691 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 32 2024-09-12T06:36:35.691 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:36:35.691 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=129, tag_tid=27, entry_tid=1], [object_number=128, tag_tid=27, entry_tid=0], [object_number=131, tag_tid=26, entry_tid=19727], [object_number=130, tag_tid=26, entry_tid=19726]]], state=connected] 2024-09-12T06:36:35.692 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=130, tag_tid=26, entry_tid=19258], [object_number=129, tag_tid=26, entry_tid=19257], [object_number=128, tag_tid=26, entry_tid=19256], [object_number=131, tag_tid=26, entry_tid=19255]]], state=connected] 2024-09-12T06:36:35.694 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:35.696 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=130, tag_tid=26, entry_tid=19258' 2024-09-12T06:36:35.697 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=129, tag_tid=27, entry_tid=1' -a 'object_number=129, tag_tid=27, entry_tid=1' = 'object_number=130, tag_tid=26, entry_tid=19258' 2024-09-12T06:36:35.697 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 'object_number=130, tag_tid=26, entry_tid=19258' '!=' 'object_number=128, tag_tid=26, entry_tid=19080' 2024-09-12T06:36:35.697 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ break 2024-09-12T06:36:35.697 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 'object_number=130, tag_tid=26, entry_tid=19258' = 'object_number=128, tag_tid=26, entry_tid=19080' 2024-09-12T06:36:35.697 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ last_mirror_pos='object_number=130, tag_tid=26, entry_tid=19258' 2024-09-12T06:36:35.697 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=129, tag_tid=27, entry_tid=1' 2024-09-12T06:36:35.697 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-12T06:36:35.697 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:36:35.697 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_tag=27 2024-09-12T06:36:35.698 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=130, tag_tid=26, entry_tid=19258' 2024-09-12T06:36:35.698 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-12T06:36:35.698 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:36:35.699 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_tag=26 2024-09-12T06:36:35.700 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=129, tag_tid=27, entry_tid=1' 2024-09-12T06:36:35.700 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-12T06:36:35.700 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:36:35.701 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_entry=1 2024-09-12T06:36:35.701 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=130, tag_tid=26, entry_tid=19258' 2024-09-12T06:36:35.701 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-12T06:36:35.701 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:36:35.702 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_entry=19258 2024-09-12T06:36:35.703 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 27 = 26 -a 1 -le 19258 2024-09-12T06:36:35.703 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:36:35.703 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:36:35.703 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:36:35.905 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror test 2024-09-12T06:36:35.905 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:36:35.905 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:36:35.905 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:36:35.905 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:36:35.905 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n test ']' 2024-09-12T06:36:35.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-12T06:36:35.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-12T06:36:35.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:36:35.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:36:35.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:36:35.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:36:35.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:36:35.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:36:35.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:36:35.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:36:35.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:36:35.909 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:36:35.909 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-12T06:36:35.909 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:36:35.909 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:36:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:36:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:36:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:36:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:36:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:36:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:36:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:36:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:36:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:36:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:36:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:36:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:36:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:36:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:36:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:36:35.911 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:36:35.911 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:36:35.911 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:36:35.911 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:36:35.911 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:36:35.911 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:36:35.911 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:36:35.911 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:36:35.911 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:36:35.911 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:36:35.911 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:36:35.911 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:36:35.911 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:36:35.927 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:36:35.927 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:36:35.927 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:36:35.927 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-12T06:36:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:36:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-12T06:36:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:36:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:36:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:36:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-12T06:36:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:36:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:36:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:36:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:36:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-12T06:36:36.017 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-test.status 2024-09-12T06:36:36.017 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:36:36.017 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:36.017 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-12T06:36:36.017 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:36.065 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 32 2024-09-12T06:36:36.065 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 32 2024-09-12T06:36:36.065 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:36:36.066 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=129, tag_tid=27, entry_tid=1], [object_number=128, tag_tid=27, entry_tid=0], [object_number=131, tag_tid=26, entry_tid=19727], [object_number=130, tag_tid=26, entry_tid=19726]]], state=connected] 2024-09-12T06:36:36.066 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=128, tag_tid=26, entry_tid=19448], [object_number=131, tag_tid=26, entry_tid=19447], [object_number=130, tag_tid=26, entry_tid=19446], [object_number=129, tag_tid=26, entry_tid=19445]]], state=connected] 2024-09-12T06:36:36.068 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:36.071 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=129, tag_tid=27, entry_tid=1' 2024-09-12T06:36:36.071 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-12T06:36:36.071 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:36:36.071 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:36:36.071 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:36:36.071 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-12T06:36:36.072 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:36:36.072 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:36:36.072 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:36:36.072 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:36:36.072 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-12T06:36:36.072 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-test.status 2024-09-12T06:36:36.072 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:36:36.073 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:36.073 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-12T06:36:36.073 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:36.120 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 32 2024-09-12T06:36:36.120 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 32 2024-09-12T06:36:36.120 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:36:36.120 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=129, tag_tid=27, entry_tid=1], [object_number=128, tag_tid=27, entry_tid=0], [object_number=131, tag_tid=26, entry_tid=19727], [object_number=130, tag_tid=26, entry_tid=19726]]], state=connected] 2024-09-12T06:36:36.120 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=128, tag_tid=26, entry_tid=19448], [object_number=131, tag_tid=26, entry_tid=19447], [object_number=130, tag_tid=26, entry_tid=19446], [object_number=129, tag_tid=26, entry_tid=19445]]], state=connected] 2024-09-12T06:36:36.123 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:36.125 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=128, tag_tid=26, entry_tid=19448' 2024-09-12T06:36:36.126 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=129, tag_tid=27, entry_tid=1' -a 'object_number=129, tag_tid=27, entry_tid=1' = 'object_number=128, tag_tid=26, entry_tid=19448' 2024-09-12T06:36:36.126 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 'object_number=128, tag_tid=26, entry_tid=19448' '!=' 'object_number=130, tag_tid=26, entry_tid=19258' 2024-09-12T06:36:36.126 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ break 2024-09-12T06:36:36.126 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 'object_number=128, tag_tid=26, entry_tid=19448' = 'object_number=130, tag_tid=26, entry_tid=19258' 2024-09-12T06:36:36.126 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ last_mirror_pos='object_number=128, tag_tid=26, entry_tid=19448' 2024-09-12T06:36:36.126 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=129, tag_tid=27, entry_tid=1' 2024-09-12T06:36:36.126 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-12T06:36:36.126 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:36:36.127 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_tag=27 2024-09-12T06:36:36.127 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=128, tag_tid=26, entry_tid=19448' 2024-09-12T06:36:36.127 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-12T06:36:36.127 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:36:36.128 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_tag=26 2024-09-12T06:36:36.128 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=129, tag_tid=27, entry_tid=1' 2024-09-12T06:36:36.129 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-12T06:36:36.129 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:36:36.129 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_entry=1 2024-09-12T06:36:36.130 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=128, tag_tid=26, entry_tid=19448' 2024-09-12T06:36:36.130 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-12T06:36:36.130 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:36:36.131 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_entry=19448 2024-09-12T06:36:36.131 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 27 = 26 -a 1 -le 19448 2024-09-12T06:36:36.131 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:36:36.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:36:36.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:36:36.333 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror test 2024-09-12T06:36:36.333 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:36:36.334 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:36:36.334 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:36:36.334 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:36:36.334 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n test ']' 2024-09-12T06:36:36.334 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-12T06:36:36.334 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-12T06:36:36.334 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:36:36.334 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:36:36.334 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:36:36.334 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:36:36.334 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:36:36.334 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:36:36.334 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:36:36.335 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:36:36.335 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:36:36.338 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:36:36.338 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-12T06:36:36.338 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:36:36.338 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:36:36.338 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:36:36.338 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:36:36.338 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:36:36.338 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:36:36.338 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:36:36.338 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:36:36.339 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:36:36.339 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:36:36.339 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:36:36.339 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:36:36.339 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:36:36.339 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:36:36.339 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:36:36.339 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:36:36.339 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:36:36.339 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:36:36.339 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:36:36.339 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:36:36.339 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:36:36.339 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:36:36.340 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:36:36.340 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:36:36.340 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:36:36.340 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:36:36.340 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:36:36.340 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:36:36.340 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:36:36.340 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:36:36.355 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:36:36.355 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:36:36.355 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:36:36.355 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-12T06:36:36.444 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:36:36.445 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-12T06:36:36.445 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:36:36.445 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:36:36.445 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:36:36.445 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-12T06:36:36.445 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:36:36.445 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:36:36.445 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:36:36.445 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:36:36.445 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-12T06:36:36.445 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-test.status 2024-09-12T06:36:36.446 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:36:36.446 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:36.446 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-12T06:36:36.446 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:36.496 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 32 2024-09-12T06:36:36.496 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 32 2024-09-12T06:36:36.496 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:36:36.496 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=129, tag_tid=27, entry_tid=1], [object_number=128, tag_tid=27, entry_tid=0], [object_number=131, tag_tid=26, entry_tid=19727], [object_number=130, tag_tid=26, entry_tid=19726]]], state=connected] 2024-09-12T06:36:36.496 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=130, tag_tid=26, entry_tid=19650], [object_number=129, tag_tid=26, entry_tid=19649], [object_number=128, tag_tid=26, entry_tid=19648], [object_number=131, tag_tid=26, entry_tid=19647]]], state=connected] 2024-09-12T06:36:36.507 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:36.508 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=129, tag_tid=27, entry_tid=1' 2024-09-12T06:36:36.509 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-12T06:36:36.509 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:36:36.509 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:36:36.509 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:36:36.509 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-12T06:36:36.509 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:36:36.509 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:36:36.509 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:36:36.510 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:36:36.510 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-12T06:36:36.510 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-test.status 2024-09-12T06:36:36.510 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:36:36.510 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:36.510 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-12T06:36:36.510 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:36.565 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 32 2024-09-12T06:36:36.565 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 32 2024-09-12T06:36:36.565 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:36:36.565 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=129, tag_tid=27, entry_tid=1], [object_number=128, tag_tid=27, entry_tid=0], [object_number=131, tag_tid=26, entry_tid=19727], [object_number=130, tag_tid=26, entry_tid=19726]]], state=connected] 2024-09-12T06:36:36.565 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=130, tag_tid=26, entry_tid=19650], [object_number=129, tag_tid=26, entry_tid=19649], [object_number=128, tag_tid=26, entry_tid=19648], [object_number=131, tag_tid=26, entry_tid=19647]]], state=connected] 2024-09-12T06:36:36.565 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:36.567 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=130, tag_tid=26, entry_tid=19650' 2024-09-12T06:36:36.567 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=129, tag_tid=27, entry_tid=1' -a 'object_number=129, tag_tid=27, entry_tid=1' = 'object_number=130, tag_tid=26, entry_tid=19650' 2024-09-12T06:36:36.567 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 'object_number=130, tag_tid=26, entry_tid=19650' '!=' 'object_number=128, tag_tid=26, entry_tid=19448' 2024-09-12T06:36:36.567 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ break 2024-09-12T06:36:36.567 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 'object_number=130, tag_tid=26, entry_tid=19650' = 'object_number=128, tag_tid=26, entry_tid=19448' 2024-09-12T06:36:36.567 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ last_mirror_pos='object_number=130, tag_tid=26, entry_tid=19650' 2024-09-12T06:36:36.567 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=129, tag_tid=27, entry_tid=1' 2024-09-12T06:36:36.567 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-12T06:36:36.567 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:36:36.568 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_tag=27 2024-09-12T06:36:36.569 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=130, tag_tid=26, entry_tid=19650' 2024-09-12T06:36:36.569 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-12T06:36:36.569 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:36:36.570 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_tag=26 2024-09-12T06:36:36.570 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=129, tag_tid=27, entry_tid=1' 2024-09-12T06:36:36.570 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-12T06:36:36.570 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:36:36.571 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_entry=1 2024-09-12T06:36:36.572 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=130, tag_tid=26, entry_tid=19650' 2024-09-12T06:36:36.572 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-12T06:36:36.572 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:36:36.573 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_entry=19650 2024-09-12T06:36:36.573 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 27 = 26 -a 1 -le 19650 2024-09-12T06:36:36.573 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:36:36.573 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:36:36.573 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:36:36.775 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror test 2024-09-12T06:36:36.775 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:36:36.775 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:36:36.776 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:36:36.776 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:36:36.776 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n test ']' 2024-09-12T06:36:36.776 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-12T06:36:36.776 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-12T06:36:36.776 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:36:36.776 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:36:36.776 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:36:36.776 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:36:36.776 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:36:36.776 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:36:36.776 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:36:36.776 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:36:36.777 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:36:36.779 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:36:36.779 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-12T06:36:36.779 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:36:36.779 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:36:36.780 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:36:36.780 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:36:36.780 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:36:36.780 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:36:36.780 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:36:36.780 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:36:36.780 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:36:36.780 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:36:36.780 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:36:36.780 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:36:36.780 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:36:36.780 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:36:36.780 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:36:36.781 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:36:36.781 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:36:36.781 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:36:36.781 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:36:36.781 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:36:36.781 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:36:36.781 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:36:36.781 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:36:36.781 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:36:36.781 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:36:36.781 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:36:36.781 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:36:36.781 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:36:36.781 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:36:36.781 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:36:36.797 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:36:36.797 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:36:36.798 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:36:36.798 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-12T06:36:36.884 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:36:36.884 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-12T06:36:36.884 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:36:36.884 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:36:36.885 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:36:36.885 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-12T06:36:36.885 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:36:36.885 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:36:36.885 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:36:36.885 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:36:36.885 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-12T06:36:36.885 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-test.status 2024-09-12T06:36:36.885 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:36:36.886 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:36.886 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-12T06:36:36.886 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:36.932 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 32 2024-09-12T06:36:36.932 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 32 2024-09-12T06:36:36.932 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:36:36.932 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=129, tag_tid=27, entry_tid=1], [object_number=128, tag_tid=27, entry_tid=0], [object_number=131, tag_tid=26, entry_tid=19727], [object_number=130, tag_tid=26, entry_tid=19726]]], state=connected] 2024-09-12T06:36:36.932 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=129, tag_tid=27, entry_tid=1], [object_number=128, tag_tid=27, entry_tid=0], [object_number=131, tag_tid=26, entry_tid=19727], [object_number=130, tag_tid=26, entry_tid=19726]]], state=connected] 2024-09-12T06:36:36.935 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:36.936 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=129, tag_tid=27, entry_tid=1' 2024-09-12T06:36:36.937 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-12T06:36:36.938 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:36:36.938 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:36:36.938 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:36:36.938 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-12T06:36:36.938 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:36:36.938 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:36:36.938 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:36:36.938 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:36:36.938 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-12T06:36:36.938 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-test.status 2024-09-12T06:36:36.938 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:36:36.939 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:36.939 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-12T06:36:36.939 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:36.986 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 32 2024-09-12T06:36:36.986 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 32 2024-09-12T06:36:36.987 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:36:36.987 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=129, tag_tid=27, entry_tid=1], [object_number=128, tag_tid=27, entry_tid=0], [object_number=131, tag_tid=26, entry_tid=19727], [object_number=130, tag_tid=26, entry_tid=19726]]], state=connected] 2024-09-12T06:36:36.987 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=129, tag_tid=27, entry_tid=1], [object_number=128, tag_tid=27, entry_tid=0], [object_number=131, tag_tid=26, entry_tid=19727], [object_number=130, tag_tid=26, entry_tid=19726]]], state=connected] 2024-09-12T06:36:36.989 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:36:36.991 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=129, tag_tid=27, entry_tid=1' 2024-09-12T06:36:36.991 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=129, tag_tid=27, entry_tid=1' -a 'object_number=129, tag_tid=27, entry_tid=1' = 'object_number=129, tag_tid=27, entry_tid=1' 2024-09-12T06:36:36.991 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:36:36.992 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_snap_present cluster1 mirror test snap9 2024-09-12T06:36:36.992 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:36:36.992 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:36:36.992 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:36:36.992 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap9 2024-09-12T06:36:36.992 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:36:36.992 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T06:36:36.992 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:36:37.992 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 info mirror/test@snap9 2024-09-12T06:36:38.046 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:rbd image 'test': 2024-09-12T06:36:38.047 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: size 512 MiB in 128 objects 2024-09-12T06:36:38.047 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: order 22 (4 MiB objects) 2024-09-12T06:36:38.047 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: snapshot_count: 17 2024-09-12T06:36:38.047 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: id: 10a8e43c53b8 2024-09-12T06:36:38.047 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: block_name_prefix: rbd_data.10a8e43c53b8 2024-09-12T06:36:38.047 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: format: 2 2024-09-12T06:36:38.047 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: features: layering, exclusive-lock, journaling 2024-09-12T06:36:38.047 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: op_features: 2024-09-12T06:36:38.048 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: flags: 2024-09-12T06:36:38.048 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: create_timestamp: Thu Sep 12 06:29:41 2024 2024-09-12T06:36:38.048 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: access_timestamp: Thu Sep 12 06:35:40 2024 2024-09-12T06:36:38.048 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: modify_timestamp: Thu Sep 12 06:36:15 2024 2024-09-12T06:36:38.048 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: protected: False 2024-09-12T06:36:38.048 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: journal: 10a8e43c53b8 2024-09-12T06:36:38.048 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring state: enabled 2024-09-12T06:36:38.048 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring mode: journal 2024-09-12T06:36:38.048 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring global id: 5bf4c9df-a9e9-4aaa-9110-4727ba332309 2024-09-12T06:36:38.048 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring primary: false 2024-09-12T06:36:38.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:36:38.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n snap8-clean ']' 2024-09-12T06:36:38.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror test snap8-clean 2024-09-12T06:36:38.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:36:38.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:36:38.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap8-clean 2024-09-12T06:36:38.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:36:38.052 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export 2024-09-12T06:36:38.052 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:36:38.052 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:36:38.052 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export 2024-09-12T06:36:38.574 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T06:36:38.585 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:36:38.886 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:36:38.888+0000 7fa3176b1640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-12T06:36:38.889 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:36:38.891+0000 7fa310ea4640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-12T06:36:38.890 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:36:38.891+0000 7fa3126a7640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-12T06:36:38.890 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:36:38.892+0000 7fa3136a9640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-12T06:36:38.891 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:36:38.893+0000 7fa311ea6640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-12T06:36:39.034 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:36:39.036+0000 7fa312ea8640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-12T06:36:39.034 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:36:39.036+0000 7fa311ea6640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-12T06:36:39.036 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:36:39.037+0000 7fa313eaa640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-12T06:36:39.258 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:36:39.260+0000 7fa317eb2640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-12T06:36:39.258 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:36:39.260+0000 7fa312ea8640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-09-12T06:36:39.378 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:36:39.379+0000 7fac02e21640 -1 bdev(0x55749a596c00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 2 2024-09-12T06:36:39.378 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:36:39.380+0000 7fabfde17640 -1 bdev(0x55749a596c00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-09-12T06:36:39.445 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T06:36:39.454 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:36:39.666 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:36:39.787 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:36:39.787 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror test snap9 2024-09-12T06:36:39.787 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:36:39.787 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:36:39.787 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap9 2024-09-12T06:36:39.787 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:36:39.787 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export 2024-09-12T06:36:39.787 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:36:39.788 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:36:39.788 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export 2024-09-12T06:36:40.316 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T06:36:40.326 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:36:41.690 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:36:41.691+0000 7fa313eaa640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-12T06:36:41.691 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:36:41.691+0000 7fa314eac640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-12T06:36:41.692 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:36:41.693+0000 7fa3176b1640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-09-12T06:36:41.818 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:36:41.818+0000 7fa315eae640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-12T06:36:41.818 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:36:41.818+0000 7fa312ea8640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-09-12T06:36:41.963 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:36:41.963+0000 7fabfe618640 -1 bdev(0x55749a596c00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 2 2024-09-12T06:36:41.963 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:36:41.964+0000 7fabfee19640 -1 bdev(0x55749a596c00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 2 2024-09-12T06:36:42.032 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:36:42.034+0000 7fa3156ad640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-12T06:36:42.162 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T06:36:42.171 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:36:42.383 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:36:42.500 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:36:42.500 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ clean_snap_name=snap9-clean 2024-09-12T06:36:42.500 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_snap cluster2 mirror test snap9-clean 2024-09-12T06:36:42.500 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:36:42.500 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:36:42.500 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:36:42.501 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap9-clean 2024-09-12T06:36:42.501 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9-clean --debug-rbd=20 --debug-journaler=20 2024-09-12T06:36:42.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 10` 2024-09-12T06:36:42.907 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ stress_write_image cluster2 mirror test 2024-09-12T06:36:42.907 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:36:42.907 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:36:42.907 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:36:42.907 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T06:36:42.908 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local duration=12 2024-09-12T06:36:42.908 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set +e 2024-09-12T06:36:42.908 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-12T06:36:44.838 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:100 IOs, 41355264 bytes 2024-09-12T06:36:47.435 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:200 IOs, 96717824 bytes 2024-09-12T06:36:48.586 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:300 IOs, 115888128 bytes 2024-09-12T06:36:50.778 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:400 IOs, 160999424 bytes 2024-09-12T06:36:51.501 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:500 IOs, 172926976 bytes 2024-09-12T06:36:52.215 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:600 IOs, 189171712 bytes 2024-09-12T06:36:52.758 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:700 IOs, 200683520 bytes 2024-09-12T06:36:53.249 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:36:53.250+0000 7fac12a52640 -1 reset not still connected to 0x55749b2be8f0 2024-09-12T06:36:53.249 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:36:53.250+0000 7fac12a52640 -1 reset not still connected to 0x55749b2bed00 2024-09-12T06:36:53.250 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:36:53.251+0000 7fac12a52640 -1 reset not still connected to 0x55749b2bedd0 2024-09-12T06:36:53.251 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:36:53.251+0000 7fac12a52640 -1 reset not still connected to 0x55749b2bf6c0 2024-09-12T06:36:54.918 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ error_code=124 2024-09-12T06:36:54.918 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set -e 2024-09-12T06:36:54.918 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T06:36:54.918 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:36:54.919 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-12T06:36:54.919 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:36:54.919 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:36:54.919 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:36:54.919 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state_pattern=up+replaying 2024-09-12T06:36:54.919 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local description_pattern= 2024-09-12T06:36:54.919 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local service_pattern= 2024-09-12T06:36:54.920 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:36:54.920 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:36:55.919 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-12T06:36:55.920 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:36:55.920 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:36:55.920 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:36:55.920 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state_pattern=up+replaying 2024-09-12T06:36:55.920 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local description_pattern= 2024-09-12T06:36:55.920 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local service_pattern= 2024-09-12T06:36:55.920 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local status 2024-09-12T06:36:55.920 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ CEPH_ARGS= 2024-09-12T06:36:55.920 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-12T06:36:55.976 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ status='test: 2024-09-12T06:36:55.976 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: global_id: 5bf4c9df-a9e9-4aaa-9110-4727ba332309 2024-09-12T06:36:55.976 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: state: up+replaying 2024-09-12T06:36:55.976 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: description: replaying, {"bytes_per_second":6396430.4,"entries_behind_primary":0,"entries_per_second":401.3,"non_primary_position":{"entry_tid":1,"object_number":129,"tag_tid":27},"primary_position":{"entry_tid":1,"object_number":129,"tag_tid":27}} 2024-09-12T06:36:55.976 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: service: mirror.0 on smithi152 2024-09-12T06:36:55.976 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: last_update: 2024-09-12 06:36:39 2024-09-12T06:36:55.976 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: peer_sites: 2024-09-12T06:36:55.976 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: name: cluster2 2024-09-12T06:36:55.977 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: state: up+stopped 2024-09-12T06:36:55.977 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: description: local image is primary 2024-09-12T06:36:55.977 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: last_update: 2024-09-12 06:36:54' 2024-09-12T06:36:55.977 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep '^ state: .*up+replaying' 2024-09-12T06:36:55.980 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: state: up+replaying 2024-09-12T06:36:55.981 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep '^ description: .*' 2024-09-12T06:36:55.981 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: description: replaying, {"bytes_per_second":6396430.4,"entries_behind_primary":0,"entries_per_second":401.3,"non_primary_position":{"entry_tid":1,"object_number":129,"tag_tid":27},"primary_position":{"entry_tid":1,"object_number":129,"tag_tid":27}} 2024-09-12T06:36:55.981 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n '' ']' 2024-09-12T06:36:55.981 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo up+replaying 2024-09-12T06:36:55.981 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep '^up+' 2024-09-12T06:36:55.982 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:up+replaying 2024-09-12T06:36:55.982 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep 'service: *mirror..* on ' 2024-09-12T06:36:55.982 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: service: mirror.0 on smithi152 2024-09-12T06:36:55.983 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local last_update 2024-09-12T06:36:55.984 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-12T06:36:55.984 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ last_update='2024-09-12 06:36:39' 2024-09-12T06:36:55.984 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-12 06:36:39' 2024-09-12T06:36:55.984 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:36:55.984 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:36:55.984 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:36:55.984 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state_pattern=up+replaying 2024-09-12T06:36:55.984 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'prev_last_update=2024-09-12 06:36:39' 2024-09-12T06:36:55.984 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local status 2024-09-12T06:36:55.984 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ CEPH_ARGS= 2024-09-12T06:36:55.985 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-12T06:36:56.043 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ status='OKOKOK142614264mirror.0smithi15219.3.0-4883-gd61b0886trueOK42684271mirror.1smithi15219.3.0-4883-gd61b0886falseOKtest5bf4c9df-a9e9-4aaa-9110-4727ba332309up+replayingreplaying, {"bytes_per_second":6396430.4,"entries_behind_primary":0,"entries_per_second":401.3,"non_primary_position":{"entry_tid":1,"object_number":129,"tag_tid":27},"primary_position":{"entry_tid":1,"object_number":129,"tag_tid":27}}42614264mirror.0smithi1522024-09-12 06:36:39cluster24b9b47ae-ca94-4dab-91d1-08bce6a93349up+stoppedlocal image is primary2024-09-12 06:36:54' 2024-09-12T06:36:56.043 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local last_update state 2024-09-12T06:36:56.043 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-12T06:36:56.045 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ last_update='2024-09-12 06:36:39' 2024-09-12T06:36:56.045 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-12T06:36:56.047 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ state=up+replaying 2024-09-12T06:36:56.047 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo up+replaying 2024-09-12T06:36:56.047 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep up+replaying 2024-09-12T06:36:56.047 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:up+replaying 2024-09-12T06:36:56.048 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:36:56.049 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:36:56.049 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ snap_name=snap10 2024-09-12T06:36:56.049 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_snap cluster2 mirror test snap10 2024-09-12T06:36:56.049 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:36:56.049 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:36:56.049 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:36:56.049 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap10 2024-09-12T06:36:56.049 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10 --debug-rbd=20 --debug-journaler=20 2024-09-12T06:37:01.541 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:37:01.542+0000 7fac12a52640 -1 reset not still connected to 0x55749c9a7860 2024-09-12T06:37:24.467 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-12T06:37:24.467 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:37:24.467 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:37:24.467 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:37:24.468 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-12T06:37:24.468 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:37:24.468 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:37:24.468 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:37:24.468 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=started 2024-09-12T06:37:24.468 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:37:24.468 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:37:24.468 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:37:25.468 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-12T06:37:25.468 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:37:25.468 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:37:25.468 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:37:25.468 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=started 2024-09-12T06:37:25.468 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local status_result 2024-09-12T06:37:25.468 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=stopped 2024-09-12T06:37:25.469 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-12T06:37:25.469 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_instance=cluster1 2024-09-12T06:37:25.469 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:37:25.469 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:37:25.469 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -i state 2024-09-12T06:37:25.469 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance=cluster1 2024-09-12T06:37:25.469 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local loop_instance 2024-09-12T06:37:25.469 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:37:25.469 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sleep 0 2024-09-12T06:37:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:37:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ seq 0 1 2024-09-12T06:37:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:37:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-12T06:37:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:0 2024-09-12T06:37:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:37:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:37:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:37:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:37:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:37:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:37:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:37:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:37:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:37:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:37:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:37:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:37:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:37:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:37:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:0 2024-09-12T06:37:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:37:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:37:25.475 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:37:25.475 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:0 2024-09-12T06:37:25.475 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:37:25.475 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:37:25.475 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:37:25.475 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:37:25.475 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T06:37:25.475 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:37:25.475 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:37:25.475 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=0 2024-09-12T06:37:25.475 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=0 2024-09-12T06:37:25.476 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:37:25.490 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:37:25.490 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:37:25.490 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:37:25.490 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-12T06:37:25.585 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ return 0 2024-09-12T06:37:25.586 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T06:37:25.586 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo ' "state": "Replaying"' 2024-09-12T06:37:25.586 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep -i Replaying 2024-09-12T06:37:25.587 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: "state": "Replaying" 2024-09-12T06:37:25.589 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_state=started 2024-09-12T06:37:25.589 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test started = started 2024-09-12T06:37:25.589 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:37:25.589 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror test 2024-09-12T06:37:25.589 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:37:25.589 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:37:25.589 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:37:25.589 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:37:25.589 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:37:25.589 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' journal = journal ']' 2024-09-12T06:37:25.590 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror test 2024-09-12T06:37:25.590 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:37:25.590 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:37:25.590 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:37:25.590 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:37:25.590 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:37:25.590 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-12T06:37:25.590 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-12T06:37:25.590 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:37:25.590 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:37:25.590 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:37:25.790 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror test 2024-09-12T06:37:25.790 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:37:25.790 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:37:25.790 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:37:25.790 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:37:25.790 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n test ']' 2024-09-12T06:37:25.790 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-12T06:37:25.790 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-12T06:37:25.791 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:37:25.791 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:37:25.791 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:37:25.791 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:37:25.791 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:37:25.791 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:37:25.791 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:37:25.791 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:37:25.791 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:37:25.794 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:37:25.794 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-12T06:37:25.794 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:37:25.794 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:37:25.794 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:37:25.794 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:37:25.795 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:37:25.795 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:37:25.795 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:37:25.795 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:37:25.795 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:37:25.795 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:37:25.795 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:37:25.795 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:37:25.795 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:37:25.795 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:37:25.795 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:37:25.795 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:37:25.795 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:37:25.795 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:37:25.795 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:37:25.796 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:37:25.796 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:37:25.796 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:37:25.796 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:37:25.796 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:37:25.796 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:37:25.796 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:37:25.796 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:37:25.796 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:37:25.796 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:37:25.796 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:37:25.812 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:37:25.812 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:37:25.812 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:37:25.812 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-12T06:37:25.897 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:37:25.897 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-12T06:37:25.897 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:37:25.897 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:37:25.897 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:37:25.897 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-12T06:37:25.897 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:37:25.898 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:37:25.898 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:37:25.898 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:37:25.898 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-12T06:37:25.898 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-test.status 2024-09-12T06:37:25.898 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:37:25.899 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:37:25.899 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-12T06:37:25.899 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:37:25.946 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 37 2024-09-12T06:37:25.946 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 37 2024-09-12T06:37:25.946 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:37:25.946 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=30, entry_tid=1], [object_number=148, tag_tid=30, entry_tid=0], [object_number=150, tag_tid=29, entry_tid=15986], [object_number=151, tag_tid=29, entry_tid=15983]]], state=connected] 2024-09-12T06:37:25.946 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=150, tag_tid=29, entry_tid=15986], [object_number=149, tag_tid=29, entry_tid=15985], [object_number=148, tag_tid=29, entry_tid=15984], [object_number=151, tag_tid=29, entry_tid=15983]]], state=connected] 2024-09-12T06:37:25.949 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:37:25.950 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=149, tag_tid=30, entry_tid=1' 2024-09-12T06:37:25.952 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-12T06:37:25.952 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:37:25.952 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:37:25.952 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:37:25.952 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-12T06:37:25.952 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:37:25.952 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:37:25.952 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:37:25.952 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:37:25.953 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-12T06:37:25.953 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-test.status 2024-09-12T06:37:25.953 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:37:25.953 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:37:25.953 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-12T06:37:25.953 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:37:26.000 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 37 2024-09-12T06:37:26.000 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 37 2024-09-12T06:37:26.001 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:37:26.001 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=30, entry_tid=1], [object_number=148, tag_tid=30, entry_tid=0], [object_number=150, tag_tid=29, entry_tid=15986], [object_number=151, tag_tid=29, entry_tid=15983]]], state=connected] 2024-09-12T06:37:26.001 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=150, tag_tid=29, entry_tid=15986], [object_number=149, tag_tid=29, entry_tid=15985], [object_number=148, tag_tid=29, entry_tid=15984], [object_number=151, tag_tid=29, entry_tid=15983]]], state=connected] 2024-09-12T06:37:26.003 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:37:26.005 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=150, tag_tid=29, entry_tid=15986' 2024-09-12T06:37:26.006 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=149, tag_tid=30, entry_tid=1' -a 'object_number=149, tag_tid=30, entry_tid=1' = 'object_number=150, tag_tid=29, entry_tid=15986' 2024-09-12T06:37:26.006 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 'object_number=150, tag_tid=29, entry_tid=15986' '!=' '' 2024-09-12T06:37:26.006 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ break 2024-09-12T06:37:26.006 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 'object_number=150, tag_tid=29, entry_tid=15986' = '' 2024-09-12T06:37:26.006 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ last_mirror_pos='object_number=150, tag_tid=29, entry_tid=15986' 2024-09-12T06:37:26.006 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=149, tag_tid=30, entry_tid=1' 2024-09-12T06:37:26.006 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-12T06:37:26.006 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:37:26.007 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_tag=30 2024-09-12T06:37:26.007 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=150, tag_tid=29, entry_tid=15986' 2024-09-12T06:37:26.007 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-12T06:37:26.007 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:37:26.008 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_tag=29 2024-09-12T06:37:26.009 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=149, tag_tid=30, entry_tid=1' 2024-09-12T06:37:26.009 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-12T06:37:26.009 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:37:26.009 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_entry=1 2024-09-12T06:37:26.010 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=150, tag_tid=29, entry_tid=15986' 2024-09-12T06:37:26.010 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-12T06:37:26.010 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:37:26.012 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_entry=15986 2024-09-12T06:37:26.012 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 30 = 29 -a 1 -le 15986 2024-09-12T06:37:26.012 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:37:26.012 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:37:26.012 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:37:26.214 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror test 2024-09-12T06:37:26.214 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:37:26.214 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:37:26.214 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:37:26.214 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:37:26.215 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n test ']' 2024-09-12T06:37:26.215 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-12T06:37:26.215 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-12T06:37:26.215 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:37:26.215 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:37:26.215 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:37:26.215 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:37:26.215 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:37:26.215 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:37:26.215 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:37:26.215 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:37:26.215 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:37:26.218 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:37:26.218 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-12T06:37:26.219 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:37:26.219 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:37:26.219 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:37:26.219 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:37:26.219 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:37:26.219 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:37:26.219 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:37:26.219 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:37:26.219 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:37:26.219 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:37:26.219 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:37:26.219 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:37:26.220 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:37:26.220 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:37:26.220 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:37:26.220 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:37:26.220 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:37:26.220 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:37:26.220 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:37:26.220 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:37:26.220 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:37:26.220 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:37:26.220 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:37:26.220 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:37:26.220 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:37:26.220 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:37:26.220 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:37:26.221 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:37:26.221 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:37:26.221 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:37:26.235 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:37:26.235 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:37:26.236 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:37:26.236 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-12T06:37:26.319 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:37:26.319 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-12T06:37:26.319 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:37:26.319 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:37:26.319 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:37:26.319 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-12T06:37:26.319 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:37:26.319 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:37:26.319 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:37:26.319 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:37:26.319 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-12T06:37:26.320 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-test.status 2024-09-12T06:37:26.320 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:37:26.320 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:37:26.320 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-12T06:37:26.320 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:37:26.371 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 37 2024-09-12T06:37:26.371 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 37 2024-09-12T06:37:26.371 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:37:26.371 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=30, entry_tid=1], [object_number=148, tag_tid=30, entry_tid=0], [object_number=150, tag_tid=29, entry_tid=15986], [object_number=151, tag_tid=29, entry_tid=15983]]], state=connected] 2024-09-12T06:37:26.371 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=150, tag_tid=29, entry_tid=15986], [object_number=149, tag_tid=29, entry_tid=15985], [object_number=148, tag_tid=29, entry_tid=15984], [object_number=151, tag_tid=29, entry_tid=15983]]], state=connected] 2024-09-12T06:37:26.374 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:37:26.375 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=149, tag_tid=30, entry_tid=1' 2024-09-12T06:37:26.376 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-12T06:37:26.376 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:37:26.376 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:37:26.377 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:37:26.377 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-12T06:37:26.377 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:37:26.377 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:37:26.377 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:37:26.377 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:37:26.377 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-12T06:37:26.377 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-test.status 2024-09-12T06:37:26.377 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:37:26.378 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:37:26.378 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-12T06:37:26.378 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:37:26.425 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 37 2024-09-12T06:37:26.425 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 37 2024-09-12T06:37:26.426 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:37:26.426 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=30, entry_tid=1], [object_number=148, tag_tid=30, entry_tid=0], [object_number=150, tag_tid=29, entry_tid=15986], [object_number=151, tag_tid=29, entry_tid=15983]]], state=connected] 2024-09-12T06:37:26.426 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=150, tag_tid=29, entry_tid=15986], [object_number=149, tag_tid=29, entry_tid=15985], [object_number=148, tag_tid=29, entry_tid=15984], [object_number=151, tag_tid=29, entry_tid=15983]]], state=connected] 2024-09-12T06:37:26.428 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:37:26.430 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=150, tag_tid=29, entry_tid=15986' 2024-09-12T06:37:26.430 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=149, tag_tid=30, entry_tid=1' -a 'object_number=149, tag_tid=30, entry_tid=1' = 'object_number=150, tag_tid=29, entry_tid=15986' 2024-09-12T06:37:26.430 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 'object_number=150, tag_tid=29, entry_tid=15986' '!=' 'object_number=150, tag_tid=29, entry_tid=15986' 2024-09-12T06:37:26.430 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:37:26.430 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.4 2024-09-12T06:37:26.836 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror test 2024-09-12T06:37:26.836 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:37:26.836 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:37:26.836 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:37:26.836 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:37:26.836 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n test ']' 2024-09-12T06:37:26.836 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-12T06:37:26.836 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-12T06:37:26.836 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:37:26.837 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:37:26.837 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:37:26.837 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:37:26.837 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:37:26.837 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:37:26.837 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:37:26.837 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:37:26.837 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:37:26.837 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:37:26.837 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-12T06:37:26.837 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:37:26.837 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:37:26.837 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:37:26.838 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:37:26.838 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:37:26.838 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:37:26.838 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:37:26.838 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:37:26.838 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:37:26.838 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:37:26.838 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:37:26.838 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:37:26.838 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:37:26.838 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:37:26.838 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:37:26.838 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:37:26.838 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:37:26.838 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:37:26.839 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:37:26.839 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:37:26.839 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:37:26.839 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:37:26.839 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:37:26.839 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:37:26.839 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:37:26.839 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:37:26.839 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:37:26.839 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:37:26.839 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:37:26.839 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:37:26.855 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:37:26.855 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:37:26.855 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:37:26.855 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-12T06:37:26.938 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:37:26.940 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-12T06:37:26.940 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:37:26.940 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:37:26.940 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:37:26.940 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-12T06:37:26.940 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:37:26.940 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:37:26.941 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:37:26.941 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:37:26.941 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-12T06:37:26.941 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-test.status 2024-09-12T06:37:26.941 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:37:26.941 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:37:26.941 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-12T06:37:26.942 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:37:26.991 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 37 2024-09-12T06:37:26.991 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 37 2024-09-12T06:37:26.991 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:37:26.991 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=30, entry_tid=1], [object_number=148, tag_tid=30, entry_tid=0], [object_number=150, tag_tid=29, entry_tid=15986], [object_number=151, tag_tid=29, entry_tid=15983]]], state=connected] 2024-09-12T06:37:26.991 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=149, tag_tid=30, entry_tid=1], [object_number=148, tag_tid=30, entry_tid=0], [object_number=150, tag_tid=29, entry_tid=15986], [object_number=151, tag_tid=29, entry_tid=15983]]], state=connected] 2024-09-12T06:37:26.994 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:37:26.995 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=149, tag_tid=30, entry_tid=1' 2024-09-12T06:37:26.997 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-12T06:37:26.997 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:37:26.997 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:37:26.997 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:37:26.997 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-12T06:37:26.997 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:37:26.997 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:37:26.997 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:37:26.997 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:37:26.997 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-12T06:37:26.997 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-test.status 2024-09-12T06:37:26.997 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:37:26.998 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:37:26.998 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-12T06:37:26.998 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:37:27.045 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 37 2024-09-12T06:37:27.045 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 37 2024-09-12T06:37:27.045 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:37:27.045 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=30, entry_tid=1], [object_number=148, tag_tid=30, entry_tid=0], [object_number=150, tag_tid=29, entry_tid=15986], [object_number=151, tag_tid=29, entry_tid=15983]]], state=connected] 2024-09-12T06:37:27.045 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=149, tag_tid=30, entry_tid=1], [object_number=148, tag_tid=30, entry_tid=0], [object_number=150, tag_tid=29, entry_tid=15986], [object_number=151, tag_tid=29, entry_tid=15983]]], state=connected] 2024-09-12T06:37:27.048 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:37:27.050 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=149, tag_tid=30, entry_tid=1' 2024-09-12T06:37:27.050 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=149, tag_tid=30, entry_tid=1' -a 'object_number=149, tag_tid=30, entry_tid=1' = 'object_number=149, tag_tid=30, entry_tid=1' 2024-09-12T06:37:27.050 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:37:27.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_snap_present cluster1 mirror test snap10 2024-09-12T06:37:27.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:37:27.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:37:27.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:37:27.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap10 2024-09-12T06:37:27.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:37:27.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T06:37:27.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:37:28.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 info mirror/test@snap10 2024-09-12T06:37:28.105 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:rbd image 'test': 2024-09-12T06:37:28.105 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: size 512 MiB in 128 objects 2024-09-12T06:37:28.105 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: order 22 (4 MiB objects) 2024-09-12T06:37:28.105 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: snapshot_count: 19 2024-09-12T06:37:28.105 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: id: 10a8e43c53b8 2024-09-12T06:37:28.105 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: block_name_prefix: rbd_data.10a8e43c53b8 2024-09-12T06:37:28.105 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: format: 2 2024-09-12T06:37:28.106 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: features: layering, exclusive-lock, journaling 2024-09-12T06:37:28.106 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: op_features: 2024-09-12T06:37:28.106 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: flags: 2024-09-12T06:37:28.106 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: create_timestamp: Thu Sep 12 06:29:41 2024 2024-09-12T06:37:28.106 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: access_timestamp: Thu Sep 12 06:36:41 2024 2024-09-12T06:37:28.106 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: modify_timestamp: Thu Sep 12 06:37:16 2024 2024-09-12T06:37:28.106 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: protected: False 2024-09-12T06:37:28.106 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: journal: 10a8e43c53b8 2024-09-12T06:37:28.106 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring state: enabled 2024-09-12T06:37:28.106 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring mode: journal 2024-09-12T06:37:28.106 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring global id: 5bf4c9df-a9e9-4aaa-9110-4727ba332309 2024-09-12T06:37:28.106 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring primary: false 2024-09-12T06:37:28.109 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:37:28.109 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n snap9-clean ']' 2024-09-12T06:37:28.110 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror test snap9-clean 2024-09-12T06:37:28.110 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:37:28.110 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:37:28.110 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap9-clean 2024-09-12T06:37:28.110 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:37:28.110 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export 2024-09-12T06:37:28.110 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:28.110 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:28.110 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export 2024-09-12T06:37:28.643 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T06:37:28.653 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:28.967 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:37:28.968+0000 7fa3176b1640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-09-12T06:37:28.967 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:37:28.969+0000 7fa310ea4640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-12T06:37:28.970 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:37:28.971+0000 7fa316eb0640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-12T06:37:28.970 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:37:28.972+0000 7fa3156ad640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-12T06:37:29.041 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:37:29.042+0000 7fa311ea6640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-12T06:37:29.041 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:37:29.042+0000 7fa3136a9640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-12T06:37:29.043 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:37:29.045+0000 7fa3176b1640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-12T06:37:29.476 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T06:37:29.483 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:29.695 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:29.814 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:37:29.814 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror test snap10 2024-09-12T06:37:29.814 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:37:29.814 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:37:29.814 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap10 2024-09-12T06:37:29.814 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:37:29.814 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export 2024-09-12T06:37:29.814 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:29.815 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:29.815 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export 2024-09-12T06:37:30.353 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T06:37:30.364 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:30.857 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:37:30.859+0000 7f80be984640 -1 bdev(0x556b0184ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 3 2024-09-12T06:37:30.858 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:37:30.859+0000 7f80bc980640 -1 bdev(0x556b0184ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 3 2024-09-12T06:37:30.931 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:37:30.933+0000 7fa313eaa640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-12T06:37:30.931 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:37:30.933+0000 7fa316eb0640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-12T06:37:30.933 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:37:30.935+0000 7fa312ea8640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-12T06:37:31.131 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T06:37:31.140 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:31.349 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:31.465 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:37:31.465 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ clean_snap_name=snap10-clean 2024-09-12T06:37:31.465 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_snap cluster2 mirror test snap10-clean 2024-09-12T06:37:31.465 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:37:31.466 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:37:31.466 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:37:31.466 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap10-clean 2024-09-12T06:37:31.466 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10-clean --debug-rbd=20 --debug-journaler=20 2024-09-12T06:37:32.480 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-12T06:37:32.480 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:37:32.480 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:37:32.480 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:37:32.480 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-12T06:37:32.481 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:37:32.481 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:37:32.481 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:37:32.481 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=started 2024-09-12T06:37:32.481 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:37:32.481 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:37:32.481 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:37:33.481 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-12T06:37:33.482 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:37:33.482 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:37:33.482 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:37:33.482 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=started 2024-09-12T06:37:33.482 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local status_result 2024-09-12T06:37:33.482 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=stopped 2024-09-12T06:37:33.483 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-12T06:37:33.483 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_instance=cluster1 2024-09-12T06:37:33.483 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:37:33.483 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -i state 2024-09-12T06:37:33.483 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:37:33.483 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance=cluster1 2024-09-12T06:37:33.483 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local loop_instance 2024-09-12T06:37:33.483 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:37:33.484 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sleep 0 2024-09-12T06:37:33.484 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:37:33.487 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ seq 0 1 2024-09-12T06:37:33.487 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:37:33.487 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-12T06:37:33.487 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:0 2024-09-12T06:37:33.487 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:37:33.487 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:37:33.487 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:37:33.487 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:37:33.487 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:37:33.487 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:37:33.488 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:37:33.488 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:37:33.488 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:37:33.488 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:37:33.488 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:37:33.488 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:37:33.488 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:37:33.488 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:37:33.488 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:0 2024-09-12T06:37:33.488 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:37:33.488 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:37:33.488 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:37:33.488 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:0 2024-09-12T06:37:33.489 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:37:33.489 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:37:33.489 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:37:33.489 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:37:33.489 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T06:37:33.489 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:37:33.489 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:37:33.489 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=0 2024-09-12T06:37:33.489 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=0 2024-09-12T06:37:33.489 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:37:33.508 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:37:33.508 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:37:33.508 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:37:33.508 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-12T06:37:33.597 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ return 0 2024-09-12T06:37:33.597 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T06:37:33.597 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo ' "state": "Replaying"' 2024-09-12T06:37:33.597 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep -i Replaying 2024-09-12T06:37:33.598 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: "state": "Replaying" 2024-09-12T06:37:33.600 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_state=started 2024-09-12T06:37:33.600 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test started = started 2024-09-12T06:37:33.600 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:37:33.600 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror test 2024-09-12T06:37:33.600 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:37:33.600 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:37:33.600 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:37:33.601 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:37:33.601 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:37:33.601 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' journal = journal ']' 2024-09-12T06:37:33.601 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror test 2024-09-12T06:37:33.601 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:37:33.601 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:37:33.601 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:37:33.601 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:37:33.601 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:37:33.601 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-12T06:37:33.601 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-12T06:37:33.601 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:37:33.601 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:37:33.602 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:37:33.801 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror test 2024-09-12T06:37:33.801 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:37:33.801 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:37:33.801 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:37:33.802 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:37:33.802 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n test ']' 2024-09-12T06:37:33.802 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-12T06:37:33.802 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-12T06:37:33.802 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:37:33.802 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:37:33.802 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:37:33.802 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:37:33.802 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:37:33.802 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:37:33.802 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:37:33.802 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:37:33.803 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:37:33.806 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:37:33.806 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-12T06:37:33.806 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:37:33.806 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:37:33.806 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:37:33.806 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:37:33.806 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:37:33.806 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:37:33.806 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:37:33.807 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:37:33.807 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:37:33.807 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:37:33.807 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:37:33.807 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:37:33.807 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:37:33.807 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:37:33.807 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:37:33.807 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:37:33.807 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:37:33.807 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:37:33.807 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:37:33.807 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:37:33.808 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:37:33.808 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:37:33.808 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:37:33.808 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:37:33.808 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:37:33.808 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:37:33.808 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:37:33.808 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:37:33.808 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:37:33.808 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:37:33.823 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:37:33.823 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:37:33.823 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:37:33.823 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-12T06:37:33.905 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:37:33.905 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-12T06:37:33.905 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:37:33.905 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:37:33.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:37:33.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-12T06:37:33.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:37:33.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:37:33.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:37:33.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:37:33.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-12T06:37:33.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-test.status 2024-09-12T06:37:33.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:37:33.907 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:37:33.907 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-12T06:37:33.907 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:37:33.954 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 37 2024-09-12T06:37:33.954 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 37 2024-09-12T06:37:33.954 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:37:33.954 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=31, entry_tid=1], [object_number=148, tag_tid=31, entry_tid=0], [object_number=150, tag_tid=29, entry_tid=15986], [object_number=151, tag_tid=29, entry_tid=15983]]], state=connected] 2024-09-12T06:37:33.954 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=149, tag_tid=30, entry_tid=1], [object_number=148, tag_tid=30, entry_tid=0], [object_number=150, tag_tid=29, entry_tid=15986], [object_number=151, tag_tid=29, entry_tid=15983]]], state=connected] 2024-09-12T06:37:33.957 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:37:33.959 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=149, tag_tid=31, entry_tid=1' 2024-09-12T06:37:33.960 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-12T06:37:33.960 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:37:33.960 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:37:33.960 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:37:33.960 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-12T06:37:33.960 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:37:33.960 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:37:33.960 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:37:33.961 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:37:33.961 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-12T06:37:33.961 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-test.status 2024-09-12T06:37:33.961 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:37:33.961 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:37:33.961 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-12T06:37:33.961 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:37:34.011 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 37 2024-09-12T06:37:34.011 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 37 2024-09-12T06:37:34.011 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:37:34.011 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=31, entry_tid=1], [object_number=148, tag_tid=31, entry_tid=0], [object_number=150, tag_tid=29, entry_tid=15986], [object_number=151, tag_tid=29, entry_tid=15983]]], state=connected] 2024-09-12T06:37:34.011 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=149, tag_tid=30, entry_tid=1], [object_number=148, tag_tid=30, entry_tid=0], [object_number=150, tag_tid=29, entry_tid=15986], [object_number=151, tag_tid=29, entry_tid=15983]]], state=connected] 2024-09-12T06:37:34.013 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:37:34.016 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=149, tag_tid=30, entry_tid=1' 2024-09-12T06:37:34.016 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=149, tag_tid=31, entry_tid=1' -a 'object_number=149, tag_tid=31, entry_tid=1' = 'object_number=149, tag_tid=30, entry_tid=1' 2024-09-12T06:37:34.016 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 'object_number=149, tag_tid=30, entry_tid=1' '!=' '' 2024-09-12T06:37:34.016 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ break 2024-09-12T06:37:34.016 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 'object_number=149, tag_tid=30, entry_tid=1' = '' 2024-09-12T06:37:34.016 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ last_mirror_pos='object_number=149, tag_tid=30, entry_tid=1' 2024-09-12T06:37:34.016 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=149, tag_tid=31, entry_tid=1' 2024-09-12T06:37:34.016 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-12T06:37:34.016 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:37:34.017 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_tag=31 2024-09-12T06:37:34.017 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=149, tag_tid=30, entry_tid=1' 2024-09-12T06:37:34.017 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-12T06:37:34.017 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:37:34.018 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_tag=30 2024-09-12T06:37:34.019 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=149, tag_tid=31, entry_tid=1' 2024-09-12T06:37:34.019 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-12T06:37:34.019 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:37:34.020 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_entry=1 2024-09-12T06:37:34.020 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=149, tag_tid=30, entry_tid=1' 2024-09-12T06:37:34.020 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-12T06:37:34.021 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:37:34.022 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_entry=1 2024-09-12T06:37:34.022 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 31 = 30 -a 1 -le 1 2024-09-12T06:37:34.022 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:37:34.022 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:37:34.022 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:37:34.224 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror test 2024-09-12T06:37:34.224 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:37:34.224 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:37:34.224 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:37:34.224 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:37:34.225 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n test ']' 2024-09-12T06:37:34.225 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-12T06:37:34.225 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-12T06:37:34.225 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:37:34.225 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:37:34.225 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:37:34.225 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:37:34.225 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:37:34.225 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:37:34.225 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:37:34.225 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:37:34.225 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:37:34.228 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:37:34.228 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-12T06:37:34.228 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:37:34.228 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:37:34.229 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:37:34.229 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:37:34.229 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:37:34.229 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:37:34.229 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:37:34.229 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:37:34.229 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:37:34.229 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:37:34.229 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:37:34.229 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:37:34.229 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:37:34.229 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:37:34.229 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:37:34.229 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:37:34.230 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:37:34.230 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:37:34.230 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:37:34.230 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:37:34.230 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:37:34.230 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:37:34.230 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:37:34.230 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:37:34.230 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:37:34.230 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:37:34.230 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:37:34.230 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:37:34.231 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:37:34.231 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:37:34.247 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:37:34.248 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:37:34.248 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:37:34.248 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-12T06:37:34.334 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:37:34.334 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-12T06:37:34.334 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:37:34.334 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:37:34.334 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:37:34.334 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-12T06:37:34.334 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:37:34.334 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:37:34.334 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:37:34.334 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:37:34.335 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-12T06:37:34.335 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-test.status 2024-09-12T06:37:34.335 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:37:34.335 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:37:34.335 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-12T06:37:34.336 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:37:34.384 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 37 2024-09-12T06:37:34.384 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 37 2024-09-12T06:37:34.384 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:37:34.384 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=31, entry_tid=1], [object_number=148, tag_tid=31, entry_tid=0], [object_number=150, tag_tid=29, entry_tid=15986], [object_number=151, tag_tid=29, entry_tid=15983]]], state=connected] 2024-09-12T06:37:34.384 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=149, tag_tid=31, entry_tid=1], [object_number=148, tag_tid=31, entry_tid=0], [object_number=150, tag_tid=29, entry_tid=15986], [object_number=151, tag_tid=29, entry_tid=15983]]], state=connected] 2024-09-12T06:37:34.425 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:37:34.426 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=149, tag_tid=31, entry_tid=1' 2024-09-12T06:37:34.427 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-12T06:37:34.428 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:37:34.428 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:37:34.428 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:37:34.428 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-12T06:37:34.428 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:37:34.428 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:37:34.428 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:37:34.428 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:37:34.428 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-12T06:37:34.428 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-test.status 2024-09-12T06:37:34.428 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:37:34.429 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:37:34.429 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-12T06:37:34.429 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:37:34.479 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 37 2024-09-12T06:37:34.479 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 37 2024-09-12T06:37:34.479 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:37:34.479 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=149, tag_tid=31, entry_tid=1], [object_number=148, tag_tid=31, entry_tid=0], [object_number=150, tag_tid=29, entry_tid=15986], [object_number=151, tag_tid=29, entry_tid=15983]]], state=connected] 2024-09-12T06:37:34.480 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=149, tag_tid=31, entry_tid=1], [object_number=148, tag_tid=31, entry_tid=0], [object_number=150, tag_tid=29, entry_tid=15986], [object_number=151, tag_tid=29, entry_tid=15983]]], state=connected] 2024-09-12T06:37:34.482 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.status 2024-09-12T06:37:34.484 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=149, tag_tid=31, entry_tid=1' 2024-09-12T06:37:34.484 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=149, tag_tid=31, entry_tid=1' -a 'object_number=149, tag_tid=31, entry_tid=1' = 'object_number=149, tag_tid=31, entry_tid=1' 2024-09-12T06:37:34.485 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:37:34.485 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_snap_present cluster1 mirror test snap10-clean 2024-09-12T06:37:34.485 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:37:34.485 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:37:34.485 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:37:34.485 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap10-clean 2024-09-12T06:37:34.485 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:37:34.485 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T06:37:34.485 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:37:35.485 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 info mirror/test@snap10-clean 2024-09-12T06:37:35.543 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:rbd image 'test': 2024-09-12T06:37:35.543 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: size 512 MiB in 128 objects 2024-09-12T06:37:35.543 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: order 22 (4 MiB objects) 2024-09-12T06:37:35.543 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: snapshot_count: 20 2024-09-12T06:37:35.543 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: id: 10a8e43c53b8 2024-09-12T06:37:35.543 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: block_name_prefix: rbd_data.10a8e43c53b8 2024-09-12T06:37:35.543 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: format: 2 2024-09-12T06:37:35.543 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: features: layering, exclusive-lock, journaling 2024-09-12T06:37:35.543 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: op_features: 2024-09-12T06:37:35.543 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: flags: 2024-09-12T06:37:35.543 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: create_timestamp: Thu Sep 12 06:29:41 2024 2024-09-12T06:37:35.544 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: access_timestamp: Thu Sep 12 06:36:41 2024 2024-09-12T06:37:35.544 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: modify_timestamp: Thu Sep 12 06:37:16 2024 2024-09-12T06:37:35.544 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: protected: False 2024-09-12T06:37:35.544 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: journal: 10a8e43c53b8 2024-09-12T06:37:35.544 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring state: enabled 2024-09-12T06:37:35.544 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring mode: journal 2024-09-12T06:37:35.544 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring global id: 5bf4c9df-a9e9-4aaa-9110-4727ba332309 2024-09-12T06:37:35.544 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring primary: false 2024-09-12T06:37:35.546 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:37:35.546 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 1 10 2024-09-12T06:37:35.548 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 10` 2024-09-12T06:37:35.548 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ snap_name=snap1 2024-09-12T06:37:35.548 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror test snap1 2024-09-12T06:37:35.549 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:37:35.549 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:37:35.549 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap1 2024-09-12T06:37:35.549 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:37:35.549 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export 2024-09-12T06:37:35.549 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:35.549 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:35.549 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export 2024-09-12T06:37:35.969 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T06:37:35.976 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:36.284 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:37:36.286+0000 7fa315eae640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-09-12T06:37:36.285 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:37:36.287+0000 7fa316eb0640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-12T06:37:36.546 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T06:37:36.553 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:36.763 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:36.878 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:37:36.879 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ snap_name=snap1-clean 2024-09-12T06:37:36.879 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror test snap1-clean 2024-09-12T06:37:36.879 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:37:36.879 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:37:36.879 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap1-clean 2024-09-12T06:37:36.879 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:37:36.879 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export 2024-09-12T06:37:36.879 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:36.879 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:36.880 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export 2024-09-12T06:37:37.306 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T06:37:37.312 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:37.844 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T06:37:37.850 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:38.060 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:38.175 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:37:38.175 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 10` 2024-09-12T06:37:38.175 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ snap_name=snap2 2024-09-12T06:37:38.175 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror test snap2 2024-09-12T06:37:38.175 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:37:38.175 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:37:38.175 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap2 2024-09-12T06:37:38.175 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:37:38.176 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export 2024-09-12T06:37:38.176 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:38.176 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:38.176 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export 2024-09-12T06:37:38.629 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T06:37:38.636 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:39.122 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:37:39.124+0000 7fa317eb2640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-12T06:37:39.123 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:37:39.124+0000 7fa312ea8640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-12T06:37:39.123 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:37:39.125+0000 7fa316eb0640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-12T06:37:39.236 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T06:37:39.243 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:39.455 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:39.573 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:37:39.573 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ snap_name=snap2-clean 2024-09-12T06:37:39.573 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror test snap2-clean 2024-09-12T06:37:39.573 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:37:39.573 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:37:39.574 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap2-clean 2024-09-12T06:37:39.574 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:37:39.574 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export 2024-09-12T06:37:39.574 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:39.574 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:39.574 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export 2024-09-12T06:37:40.029 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T06:37:40.038 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:40.466 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:37:40.468+0000 7fa312ea8640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-09-12T06:37:40.593 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T06:37:40.600 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:40.812 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:40.927 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:37:40.927 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 10` 2024-09-12T06:37:40.927 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ snap_name=snap3 2024-09-12T06:37:40.927 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror test snap3 2024-09-12T06:37:40.927 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:37:40.927 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:37:40.927 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap3 2024-09-12T06:37:40.928 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:37:40.928 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export 2024-09-12T06:37:40.928 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:40.928 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:40.928 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export 2024-09-12T06:37:41.390 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T06:37:41.400 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:42.072 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T06:37:42.079 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:42.289 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:42.406 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:37:42.406 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ snap_name=snap3-clean 2024-09-12T06:37:42.406 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror test snap3-clean 2024-09-12T06:37:42.407 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:37:42.407 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:37:42.407 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap3-clean 2024-09-12T06:37:42.407 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:37:42.407 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export 2024-09-12T06:37:42.407 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:42.407 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:42.407 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export 2024-09-12T06:37:42.874 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T06:37:42.883 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:43.428 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:37:43.430+0000 7fa313eaa640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-09-12T06:37:43.428 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:37:43.430+0000 7fa317eb2640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-09-12T06:37:43.429 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:37:43.430+0000 7fa3156ad640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-12T06:37:43.556 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T06:37:43.563 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:43.776 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:43.896 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:37:43.896 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 10` 2024-09-12T06:37:43.896 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ snap_name=snap4 2024-09-12T06:37:43.896 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror test snap4 2024-09-12T06:37:43.897 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:37:43.897 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:37:43.897 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap4 2024-09-12T06:37:43.897 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:37:43.897 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export 2024-09-12T06:37:43.897 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:43.897 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:43.897 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export 2024-09-12T06:37:44.368 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T06:37:44.377 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:44.809 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:37:44.810+0000 7f80bf185640 -1 bdev(0x556b0184ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 1 2024-09-12T06:37:44.874 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:37:44.876+0000 7fac02620640 -1 bdev(0x55749a596c00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 3 2024-09-12T06:37:45.128 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T06:37:45.137 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:45.349 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:45.472 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:37:45.472 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ snap_name=snap4-clean 2024-09-12T06:37:45.472 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror test snap4-clean 2024-09-12T06:37:45.472 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:37:45.472 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:37:45.472 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap4-clean 2024-09-12T06:37:45.473 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:37:45.473 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export 2024-09-12T06:37:45.473 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:45.473 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:45.473 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export 2024-09-12T06:37:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T06:37:45.977 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:46.290 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:37:46.292+0000 7fa3136a9640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-12T06:37:46.750 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T06:37:46.758 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:46.970 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:47.087 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:37:47.088 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 10` 2024-09-12T06:37:47.088 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ snap_name=snap5 2024-09-12T06:37:47.088 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror test snap5 2024-09-12T06:37:47.088 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:37:47.088 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:37:47.088 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap5 2024-09-12T06:37:47.088 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:37:47.088 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export 2024-09-12T06:37:47.088 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:47.088 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:47.088 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export 2024-09-12T06:37:47.592 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T06:37:47.603 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:47.781 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:37:47.783+0000 7fa3126a7640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-09-12T06:37:47.781 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:37:47.783+0000 7fa3166af640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-12T06:37:47.925 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:37:47.927+0000 7fa311ea6640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-12T06:37:47.926 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:37:47.928+0000 7fa3126a7640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-12T06:37:47.928 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:37:47.929+0000 7fa314eac640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-12T06:37:48.052 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:37:48.054+0000 7fa317eb2640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-12T06:37:48.053 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:37:48.054+0000 7fa311ea6640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-12T06:37:48.053 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:37:48.055+0000 7fa316eb0640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-12T06:37:48.053 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:37:48.055+0000 7fa312ea8640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-12T06:37:48.117 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:37:48.119+0000 7f80c318d640 -1 bdev(0x556b0184ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 3 2024-09-12T06:37:48.195 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:37:48.197+0000 7f80be984640 -1 bdev(0x556b0184ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 2 2024-09-12T06:37:48.197 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:37:48.198+0000 7f80c1189640 -1 bdev(0x556b0184ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 4 2024-09-12T06:37:48.427 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T06:37:48.437 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:48.648 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:48.764 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:37:48.765 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ snap_name=snap5-clean 2024-09-12T06:37:48.765 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror test snap5-clean 2024-09-12T06:37:48.765 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:37:48.765 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:37:48.765 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap5-clean 2024-09-12T06:37:48.765 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:37:48.765 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export 2024-09-12T06:37:48.765 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:48.765 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:48.765 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export 2024-09-12T06:37:49.257 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T06:37:49.267 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:49.540 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:37:49.542+0000 7fa310ea4640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-12T06:37:49.541 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:37:49.542+0000 7fa3136a9640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-12T06:37:49.542 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:37:49.544+0000 7fa3166af640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-12T06:37:49.659 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:37:49.660+0000 7fa3126a7640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-12T06:37:49.660 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:37:49.661+0000 7fa316eb0640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-12T06:37:49.660 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:37:49.661+0000 7fa315eae640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-12T06:37:49.661 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:37:49.663+0000 7fa311ea6640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-12T06:37:49.661 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:37:49.663+0000 7fa313eaa640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-12T06:37:49.731 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:37:49.733+0000 7f80c318d640 -1 bdev(0x556b0184ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 1 2024-09-12T06:37:49.797 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:37:49.799+0000 7fac02620640 -1 bdev(0x55749a596c00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 2 2024-09-12T06:37:49.799 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:37:49.801+0000 7fabfe618640 -1 bdev(0x55749a596c00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2024-09-12T06:37:49.868 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:37:49.870+0000 7fa316eb0640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-12T06:37:49.870 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:37:49.871+0000 7fa312ea8640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-12T06:37:50.031 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T06:37:50.041 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:50.253 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:50.375 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:37:50.375 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 10` 2024-09-12T06:37:50.375 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ snap_name=snap6 2024-09-12T06:37:50.375 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror test snap6 2024-09-12T06:37:50.375 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:37:50.375 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:37:50.375 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap6 2024-09-12T06:37:50.376 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:37:50.376 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export 2024-09-12T06:37:50.376 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:50.376 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:50.376 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export 2024-09-12T06:37:50.881 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T06:37:50.892 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:51.529 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:37:51.530+0000 7fa312ea8640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-12T06:37:51.530 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:37:51.532+0000 7fa310ea4640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-12T06:37:51.729 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T06:37:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:51.950 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:52.068 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:37:52.068 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ snap_name=snap6-clean 2024-09-12T06:37:52.068 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror test snap6-clean 2024-09-12T06:37:52.068 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:37:52.069 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:37:52.069 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap6-clean 2024-09-12T06:37:52.069 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:37:52.069 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export 2024-09-12T06:37:52.069 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:52.069 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:52.069 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export 2024-09-12T06:37:52.583 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T06:37:52.593 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:52.888 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:37:52.890+0000 7fa3166af640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-12T06:37:52.889 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:37:52.891+0000 7fa315eae640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-12T06:37:52.889 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:37:52.891+0000 7fa3136a9640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-12T06:37:52.891 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:37:52.893+0000 7fa314eac640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-12T06:37:53.012 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:37:53.014+0000 7fa315eae640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-12T06:37:53.013 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:37:53.015+0000 7fa317eb2640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-12T06:37:53.216 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:37:53.218+0000 7fa313eaa640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-09-12T06:37:53.343 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T06:37:53.352 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:53.567 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:53.684 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:37:53.685 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 10` 2024-09-12T06:37:53.685 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ snap_name=snap7 2024-09-12T06:37:53.685 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror test snap7 2024-09-12T06:37:53.685 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:37:53.685 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:37:53.685 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap7 2024-09-12T06:37:53.685 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:37:53.685 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export 2024-09-12T06:37:53.685 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:53.685 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:53.685 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export 2024-09-12T06:37:54.207 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T06:37:54.218 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:54.279 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:37:54.280+0000 7fa313eaa640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-09-12T06:37:54.279 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:37:54.281+0000 7fa3126a7640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-12T06:37:54.502 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:37:54.504+0000 7fa3176b1640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-09-12T06:37:54.504 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:37:54.506+0000 7fa3136a9640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-12T06:37:54.504 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:37:54.506+0000 7fa310ea4640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-12T06:37:54.504 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:37:54.506+0000 7fa3166af640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-12T06:37:54.850 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:37:54.852+0000 7fa313eaa640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-12T06:37:55.048 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T06:37:55.058 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:55.268 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:55.387 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:37:55.387 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ snap_name=snap7-clean 2024-09-12T06:37:55.387 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror test snap7-clean 2024-09-12T06:37:55.387 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:37:55.387 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:37:55.387 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap7-clean 2024-09-12T06:37:55.388 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:37:55.388 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export 2024-09-12T06:37:55.388 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:55.388 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:55.388 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export 2024-09-12T06:37:55.911 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T06:37:55.921 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:56.251 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:37:56.253+0000 7fa3176b1640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-12T06:37:56.252 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:37:56.254+0000 7fa3136a9640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-12T06:37:56.718 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T06:37:56.728 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:56.939 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:57.056 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:37:57.056 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 10` 2024-09-12T06:37:57.056 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ snap_name=snap8 2024-09-12T06:37:57.056 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror test snap8 2024-09-12T06:37:57.056 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:37:57.056 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:37:57.056 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap8 2024-09-12T06:37:57.056 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:37:57.056 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export 2024-09-12T06:37:57.056 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:57.056 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:57.057 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export 2024-09-12T06:37:57.587 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T06:37:57.600 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:57.965 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:37:57.967+0000 7fa311ea6640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-12T06:37:57.966 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:37:57.968+0000 7fa3136a9640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-12T06:37:58.072 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:37:58.073+0000 7fa312ea8640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-09-12T06:37:58.227 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:37:58.229+0000 7f80c298c640 -1 bdev(0x556b0184ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 4 2024-09-12T06:37:58.302 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:37:58.304+0000 7fa317eb2640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-12T06:37:58.417 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:37:58.419+0000 7f80be183640 -1 bdev(0x556b0184ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 1 2024-09-12T06:37:58.477 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T06:37:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:58.699 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:58.821 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:37:58.821 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ snap_name=snap8-clean 2024-09-12T06:37:58.821 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror test snap8-clean 2024-09-12T06:37:58.821 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:37:58.821 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:37:58.821 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap8-clean 2024-09-12T06:37:58.821 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:37:58.822 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export 2024-09-12T06:37:58.822 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:58.822 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:58.822 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export 2024-09-12T06:37:59.344 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T06:37:59.356 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:37:59.715 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:37:59.717+0000 7fa3176b1640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-12T06:37:59.716 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:37:59.718+0000 7fa310ea4640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-12T06:37:59.801 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:37:59.803+0000 7fa313eaa640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-09-12T06:38:00.032 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:38:00.034+0000 7fa313eaa640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-12T06:38:00.033 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:38:00.034+0000 7fa316eb0640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-09-12T06:38:00.034 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:38:00.036+0000 7fa312ea8640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-09-12T06:38:00.034 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:38:00.036+0000 7fa317eb2640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-12T06:38:00.222 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T06:38:00.231 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:38:00.444 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:38:00.562 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:38:00.562 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 10` 2024-09-12T06:38:00.563 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ snap_name=snap9 2024-09-12T06:38:00.563 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror test snap9 2024-09-12T06:38:00.563 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:38:00.563 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:38:00.563 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap9 2024-09-12T06:38:00.563 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:38:00.563 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export 2024-09-12T06:38:00.563 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:38:00.563 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:38:00.563 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export 2024-09-12T06:38:01.089 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T06:38:01.101 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:38:01.391 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:38:01.393+0000 7fa3146ab640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-09-12T06:38:01.392 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:38:01.393+0000 7fa3136a9640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-12T06:38:01.392 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:38:01.394+0000 7fa3156ad640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-12T06:38:01.468 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:38:01.470+0000 7fa317eb2640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-09-12T06:38:01.469 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:38:01.471+0000 7fa3136a9640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-12T06:38:01.566 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:38:01.568+0000 7fa317eb2640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-09-12T06:38:01.566 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:38:01.568+0000 7fa312ea8640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-12T06:38:01.638 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:38:01.640+0000 7f80be984640 -1 bdev(0x556b0184ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 2 2024-09-12T06:38:01.974 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T06:38:01.983 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:38:02.194 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:38:02.311 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:38:02.311 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ snap_name=snap9-clean 2024-09-12T06:38:02.311 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror test snap9-clean 2024-09-12T06:38:02.311 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:38:02.311 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:38:02.311 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap9-clean 2024-09-12T06:38:02.311 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:38:02.311 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export 2024-09-12T06:38:02.311 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:38:02.312 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:38:02.312 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export 2024-09-12T06:38:02.860 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T06:38:02.871 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:38:03.245 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:38:03.247+0000 7fa311ea6640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-12T06:38:03.245 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:38:03.247+0000 7fa3166af640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-09-12T06:38:03.246 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:38:03.247+0000 7fa313eaa640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-12T06:38:03.247 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:38:03.249+0000 7fa316eb0640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-09-12T06:38:03.471 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:38:03.472+0000 7fa312ea8640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-12T06:38:03.642 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T06:38:03.650 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:38:03.862 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:38:03.980 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:38:03.980 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 10` 2024-09-12T06:38:03.980 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ snap_name=snap10 2024-09-12T06:38:03.981 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror test snap10 2024-09-12T06:38:03.981 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:38:03.981 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:38:03.981 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap10 2024-09-12T06:38:03.981 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:38:03.981 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export 2024-09-12T06:38:03.981 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:38:03.981 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:38:03.981 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export 2024-09-12T06:38:04.713 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T06:38:04.725 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:38:05.279 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:38:05.281+0000 7f80c1189640 -1 bdev(0x556b0184ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 3 2024-09-12T06:38:05.280 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:38:05.282+0000 7f80c298c640 -1 bdev(0x556b0184ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 4 2024-09-12T06:38:05.352 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:38:05.354+0000 7fa316eb0640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-12T06:38:05.356 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:38:05.358+0000 7fa312ea8640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-12T06:38:05.357 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:38:05.359+0000 7fa317eb2640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-09-12T06:38:05.555 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T06:38:05.566 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:38:05.778 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:38:05.897 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:38:05.897 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ snap_name=snap10-clean 2024-09-12T06:38:05.897 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror test snap10-clean 2024-09-12T06:38:05.897 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:38:05.897 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:38:05.897 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap10-clean 2024-09-12T06:38:05.897 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:38:05.897 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export 2024-09-12T06:38:05.898 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:38:05.898 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:38:05.898 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10-clean /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export 2024-09-12T06:38:06.432 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T06:38:06.444 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10-clean /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:38:06.606 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:38:06.608+0000 7fa317eb2640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-12T06:38:06.766 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:38:06.768+0000 7fa310ea4640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-09-12T06:38:06.767 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:38:06.769+0000 7fa3126a7640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-12T06:38:06.768 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:38:06.770+0000 7fa314eac640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-09-12T06:38:06.880 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:38:06.881+0000 7fa316eb0640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-12T06:38:07.202 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:38:07.204+0000 7fac03622640 -1 bdev(0x55749a596c00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-09-12T06:38:07.262 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T06:38:07.270 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:38:07.480 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-test.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-test.export 2024-09-12T06:38:07.598 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:38:07.598 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 1 10 2024-09-12T06:38:07.599 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 10` 2024-09-12T06:38:07.599 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ snap_name=snap1 2024-09-12T06:38:07.599 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_snapshot cluster2 mirror test snap1 2024-09-12T06:38:07.599 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:38:07.600 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:38:07.600 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:38:07.600 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap=snap1 2024-09-12T06:38:07.600 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1 2024-09-12T06:38:08.539 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing snap: 100% complete...done. 2024-09-12T06:38:08.572 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ snap_name=snap1-clean 2024-09-12T06:38:08.572 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_snapshot cluster2 mirror test snap1-clean 2024-09-12T06:38:08.572 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:38:08.572 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:38:08.572 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:38:08.572 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap=snap1-clean 2024-09-12T06:38:08.572 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1-clean 2024-09-12T06:38:09.537 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing snap: 100% complete...done. 2024-09-12T06:38:09.572 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 10` 2024-09-12T06:38:09.572 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ snap_name=snap2 2024-09-12T06:38:09.572 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_snapshot cluster2 mirror test snap2 2024-09-12T06:38:09.572 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:38:09.572 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:38:09.572 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:38:09.572 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap=snap2 2024-09-12T06:38:09.572 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2 2024-09-12T06:38:10.545 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing snap: 100% complete...done. 2024-09-12T06:38:10.581 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ snap_name=snap2-clean 2024-09-12T06:38:10.581 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_snapshot cluster2 mirror test snap2-clean 2024-09-12T06:38:10.581 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:38:10.581 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:38:10.581 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:38:10.581 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap=snap2-clean 2024-09-12T06:38:10.581 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2-clean 2024-09-12T06:38:11.548 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing snap: 100% complete...done. 2024-09-12T06:38:11.583 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 10` 2024-09-12T06:38:11.583 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ snap_name=snap3 2024-09-12T06:38:11.583 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_snapshot cluster2 mirror test snap3 2024-09-12T06:38:11.583 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:38:11.584 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:38:11.584 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:38:11.584 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap=snap3 2024-09-12T06:38:11.584 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3 2024-09-12T06:38:12.560 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing snap: 100% complete...done. 2024-09-12T06:38:12.589 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ snap_name=snap3-clean 2024-09-12T06:38:12.590 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_snapshot cluster2 mirror test snap3-clean 2024-09-12T06:38:12.590 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:38:12.590 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:38:12.590 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:38:12.590 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap=snap3-clean 2024-09-12T06:38:12.591 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3-clean 2024-09-12T06:38:13.556 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing snap: 100% complete...done. 2024-09-12T06:38:13.594 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 10` 2024-09-12T06:38:13.594 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ snap_name=snap4 2024-09-12T06:38:13.595 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_snapshot cluster2 mirror test snap4 2024-09-12T06:38:13.595 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:38:13.595 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:38:13.595 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:38:13.595 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap=snap4 2024-09-12T06:38:13.595 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4 2024-09-12T06:38:14.555 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing snap: 100% complete...done. 2024-09-12T06:38:14.599 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ snap_name=snap4-clean 2024-09-12T06:38:14.599 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_snapshot cluster2 mirror test snap4-clean 2024-09-12T06:38:14.599 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:38:14.599 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:38:14.599 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:38:14.599 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap=snap4-clean 2024-09-12T06:38:14.599 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4-clean 2024-09-12T06:38:14.922 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing snap: 100% complete...done. 2024-09-12T06:38:14.956 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 10` 2024-09-12T06:38:14.957 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ snap_name=snap5 2024-09-12T06:38:14.957 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_snapshot cluster2 mirror test snap5 2024-09-12T06:38:14.957 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:38:14.957 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:38:14.957 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:38:14.957 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap=snap5 2024-09-12T06:38:14.957 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5 2024-09-12T06:38:15.921 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing snap: 100% complete...done. 2024-09-12T06:38:15.950 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ snap_name=snap5-clean 2024-09-12T06:38:15.950 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_snapshot cluster2 mirror test snap5-clean 2024-09-12T06:38:15.950 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:38:15.950 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:38:15.950 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:38:15.950 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap=snap5-clean 2024-09-12T06:38:15.950 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5-clean 2024-09-12T06:38:16.923 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing snap: 100% complete...done. 2024-09-12T06:38:16.984 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 10` 2024-09-12T06:38:16.984 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ snap_name=snap6 2024-09-12T06:38:16.984 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_snapshot cluster2 mirror test snap6 2024-09-12T06:38:16.984 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:38:16.985 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:38:16.985 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:38:16.985 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap=snap6 2024-09-12T06:38:16.985 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6 2024-09-12T06:38:17.912 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing snap: 100% complete...done. 2024-09-12T06:38:17.940 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ snap_name=snap6-clean 2024-09-12T06:38:17.941 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_snapshot cluster2 mirror test snap6-clean 2024-09-12T06:38:17.941 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:38:17.941 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:38:17.941 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:38:17.941 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap=snap6-clean 2024-09-12T06:38:17.941 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6-clean 2024-09-12T06:38:18.923 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing snap: 100% complete...done. 2024-09-12T06:38:18.955 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 10` 2024-09-12T06:38:18.955 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ snap_name=snap7 2024-09-12T06:38:18.955 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_snapshot cluster2 mirror test snap7 2024-09-12T06:38:18.955 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:38:18.955 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:38:18.955 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:38:18.955 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap=snap7 2024-09-12T06:38:18.955 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7 2024-09-12T06:38:19.917 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing snap: 100% complete...done. 2024-09-12T06:38:19.943 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ snap_name=snap7-clean 2024-09-12T06:38:19.943 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_snapshot cluster2 mirror test snap7-clean 2024-09-12T06:38:19.943 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:38:19.943 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:38:19.943 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:38:19.943 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap=snap7-clean 2024-09-12T06:38:19.943 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7-clean 2024-09-12T06:38:20.945 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing snap: 100% complete...done. 2024-09-12T06:38:20.980 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 10` 2024-09-12T06:38:20.981 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ snap_name=snap8 2024-09-12T06:38:20.981 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_snapshot cluster2 mirror test snap8 2024-09-12T06:38:20.981 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:38:20.981 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:38:20.981 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:38:20.981 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap=snap8 2024-09-12T06:38:20.981 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8 2024-09-12T06:38:21.941 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing snap: 100% complete...done. 2024-09-12T06:38:21.976 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ snap_name=snap8-clean 2024-09-12T06:38:21.976 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_snapshot cluster2 mirror test snap8-clean 2024-09-12T06:38:21.976 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:38:21.976 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:38:21.977 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:38:21.977 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap=snap8-clean 2024-09-12T06:38:21.977 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8-clean 2024-09-12T06:38:22.947 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing snap: 100% complete...done. 2024-09-12T06:38:22.992 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 10` 2024-09-12T06:38:22.992 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ snap_name=snap9 2024-09-12T06:38:22.992 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_snapshot cluster2 mirror test snap9 2024-09-12T06:38:22.992 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:38:22.992 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:38:22.992 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:38:22.992 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap=snap9 2024-09-12T06:38:22.992 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9 2024-09-12T06:38:23.944 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing snap: 100% complete...done. 2024-09-12T06:38:23.971 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ snap_name=snap9-clean 2024-09-12T06:38:23.972 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_snapshot cluster2 mirror test snap9-clean 2024-09-12T06:38:23.972 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:38:23.972 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:38:23.972 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:38:23.972 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap=snap9-clean 2024-09-12T06:38:23.972 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9-clean 2024-09-12T06:38:24.924 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing snap: 100% complete...done. 2024-09-12T06:38:24.951 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 10` 2024-09-12T06:38:24.951 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ snap_name=snap10 2024-09-12T06:38:24.951 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_snapshot cluster2 mirror test snap10 2024-09-12T06:38:24.951 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:38:24.951 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:38:24.951 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:38:24.951 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap=snap10 2024-09-12T06:38:24.952 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10 2024-09-12T06:38:25.956 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing snap: 100% complete...done. 2024-09-12T06:38:25.972 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ snap_name=snap10-clean 2024-09-12T06:38:25.972 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_snapshot cluster2 mirror test snap10-clean 2024-09-12T06:38:25.972 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:38:25.973 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:38:25.973 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:38:25.973 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap=snap10-clean 2024-09-12T06:38:25.973 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10-clean 2024-09-12T06:38:26.957 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing snap: 100% complete...done. 2024-09-12T06:38:26.973 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image_retry cluster2 mirror test 2024-09-12T06:38:26.973 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:38:26.973 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:38:26.974 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:38:26.974 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T06:38:26.974 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:38:26.976 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image cluster2 mirror test 2024-09-12T06:38:26.976 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:38:26.976 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:38:26.976 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:38:26.977 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 snap purge mirror/test 2024-09-12T06:38:27.022 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 rm mirror/test 2024-09-12T06:38:27.294 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-12T06:38:27.300 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:38:27.300 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_present cluster1 mirror test deleted 2024-09-12T06:38:27.300 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:38:27.300 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:38:27.300 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:38:27.301 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=deleted 2024-09-12T06:38:27.301 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:38:27.301 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:38:27.301 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:38:27.301 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror test 2024-09-12T06:38:27.301 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:38:27.302 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:38:27.302 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:38:27.302 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-09-12T06:38:27.302 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:38:27.352 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image_id=10a8e43c53b8 2024-09-12T06:38:27.353 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T06:38:27.353 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.1 2024-09-12T06:38:27.454 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_present cluster1 mirror test deleted 10a8e43c53b8 2024-09-12T06:38:27.454 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:38:27.455 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:38:27.455 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:38:27.455 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=deleted 2024-09-12T06:38:27.455 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id=10a8e43c53b8 2024-09-12T06:38:27.455 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=deleted 2024-09-12T06:38:27.455 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_image_id 2024-09-12T06:38:27.455 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror test 2024-09-12T06:38:27.455 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:38:27.455 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:38:27.455 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:38:27.455 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-09-12T06:38:27.455 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:38:27.506 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_image_id=10a8e43c53b8 2024-09-12T06:38:27.506 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 10a8e43c53b8 2024-09-12T06:38:27.507 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -z 10a8e43c53b8 -o 10a8e43c53b8 = 10a8e43c53b8 2024-09-12T06:38:27.507 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_state=present 2024-09-12T06:38:27.507 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test deleted = present 2024-09-12T06:38:27.507 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T06:38:27.507 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:38:28.508 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_present cluster1 mirror test deleted 10a8e43c53b8 2024-09-12T06:38:28.508 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:38:28.508 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:38:28.508 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=test 2024-09-12T06:38:28.508 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=deleted 2024-09-12T06:38:28.508 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id=10a8e43c53b8 2024-09-12T06:38:28.508 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=deleted 2024-09-12T06:38:28.508 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_image_id 2024-09-12T06:38:28.509 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror test 2024-09-12T06:38:28.509 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:38:28.509 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:38:28.509 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=test 2024-09-12T06:38:28.509 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-09-12T06:38:28.509 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:38:28.548 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image test: (2) No such file or directory 2024-09-12T06:38:28.552 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_image_id= 2024-09-12T06:38:28.552 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:38:28.552 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test deleted = deleted 2024-09-12T06:38:28.552 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:38:28.553 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ testlog 'TEST: create many images' 2024-09-12T06:38:28.553 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ tee -a /tmp/tmp.GzjHZGs9Z0/rbd-mirror.test.log 2024-09-12T06:38:28.553 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ date '+%F %T' 2024-09-12T06:38:28.553 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo 2024-09-12 06:38:28 TEST: create many images 2024-09-12T06:38:28.553 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:2024-09-12 06:38:28 TEST: create many images 2024-09-12T06:38:28.554 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ snap_name=snap 2024-09-12T06:38:28.554 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 1 50 2024-09-12T06:38:28.555 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:38:28.555 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_1 2024-09-12T06:38:28.555 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image_and_enable_mirror cluster2 mirror image_1 journal 128M 2024-09-12T06:38:28.555 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:38:28.555 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:38:28.555 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:38:28.555 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:38:28.555 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_1 2024-09-12T06:38:28.555 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:38:28.556 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local mode=journal 2024-09-12T06:38:28.556 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n journal ']' 2024-09-12T06:38:28.556 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:38:28.556 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image cluster2 mirror image_1 128M 2024-09-12T06:38:28.556 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:38:28.556 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:38:28.556 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:38:28.557 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:38:28.557 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_1 2024-09-12T06:38:28.557 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:38:28.557 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local size=128 2024-09-12T06:38:28.557 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n 128M ']' 2024-09-12T06:38:28.557 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ size=128M 2024-09-12T06:38:28.557 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:38:28.557 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_1 2024-09-12T06:38:28.614 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T06:38:28.614 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:38:28.614 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' pool = image ']' 2024-09-12T06:38:28.614 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ case "${pool}" in 2024-09-12T06:38:28.614 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 1 2024-09-12T06:38:28.614 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n '' ']' 2024-09-12T06:38:28.614 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ stress_write_image cluster2 mirror image_1 2024-09-12T06:38:28.614 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:38:28.614 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:38:28.614 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_1 2024-09-12T06:38:28.614 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T06:38:28.615 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local duration=12 2024-09-12T06:38:28.615 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set +e 2024-09-12T06:38:28.615 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_1 --debug-rbd=20 --debug-journaler=20 2024-09-12T06:38:31.956 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:100 IOs, 52776960 bytes 2024-09-12T06:38:33.096 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:200 IOs, 77338624 bytes 2024-09-12T06:38:34.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:300 IOs, 98299904 bytes 2024-09-12T06:38:36.236 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:400 IOs, 150785024 bytes 2024-09-12T06:38:38.442 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:500 IOs, 172432384 bytes 2024-09-12T06:38:39.566 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:600 IOs, 193953792 bytes 2024-09-12T06:38:40.626 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ error_code=124 2024-09-12T06:38:40.626 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set -e 2024-09-12T06:38:40.626 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T06:38:40.626 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:38:40.626 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:38:40.626 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_2 2024-09-12T06:38:40.626 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image_and_enable_mirror cluster2 mirror image_2 journal 128M 2024-09-12T06:38:40.626 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:38:40.626 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:38:40.626 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:38:40.627 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:38:40.627 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_2 2024-09-12T06:38:40.627 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:38:40.627 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local mode=journal 2024-09-12T06:38:40.627 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n journal ']' 2024-09-12T06:38:40.627 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:38:40.627 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image cluster2 mirror image_2 128M 2024-09-12T06:38:40.627 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:38:40.627 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:38:40.627 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:38:40.627 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:38:40.627 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_2 2024-09-12T06:38:40.627 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:38:40.627 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local size=128 2024-09-12T06:38:40.627 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n 128M ']' 2024-09-12T06:38:40.628 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ size=128M 2024-09-12T06:38:40.628 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:38:40.628 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_2 2024-09-12T06:38:40.689 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T06:38:40.689 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:38:40.689 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' pool = image ']' 2024-09-12T06:38:40.689 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ case "${pool}" in 2024-09-12T06:38:40.689 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 1 2024-09-12T06:38:40.690 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n '' ']' 2024-09-12T06:38:40.690 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ stress_write_image cluster2 mirror image_2 2024-09-12T06:38:40.690 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:38:40.690 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:38:40.690 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_2 2024-09-12T06:38:40.690 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T06:38:40.690 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local duration=6 2024-09-12T06:38:40.691 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set +e 2024-09-12T06:38:40.691 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_2 --debug-rbd=20 --debug-journaler=20 2024-09-12T06:38:42.277 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:100 IOs, 41380864 bytes 2024-09-12T06:38:43.991 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:200 IOs, 73131008 bytes 2024-09-12T06:38:45.882 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:300 IOs, 114954240 bytes 2024-09-12T06:38:46.701 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ error_code=124 2024-09-12T06:38:46.701 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set -e 2024-09-12T06:38:46.701 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T06:38:46.701 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:38:46.701 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:38:46.701 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_3 2024-09-12T06:38:46.701 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image_and_enable_mirror cluster2 mirror image_3 journal 128M 2024-09-12T06:38:46.701 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:38:46.701 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:38:46.702 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:38:46.702 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:38:46.702 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_3 2024-09-12T06:38:46.702 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:38:46.702 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local mode=journal 2024-09-12T06:38:46.702 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n journal ']' 2024-09-12T06:38:46.702 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:38:46.702 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image cluster2 mirror image_3 128M 2024-09-12T06:38:46.702 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:38:46.702 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:38:46.702 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:38:46.702 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:38:46.702 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_3 2024-09-12T06:38:46.702 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:38:46.702 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local size=128 2024-09-12T06:38:46.702 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n 128M ']' 2024-09-12T06:38:46.703 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ size=128M 2024-09-12T06:38:46.703 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:38:46.703 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_3 2024-09-12T06:38:46.722 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:38:46.724+0000 7f80d15bb640 -1 reset not still connected to 0x556b02ee8c30 2024-09-12T06:38:46.766 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T06:38:46.766 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:38:46.766 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' pool = image ']' 2024-09-12T06:38:46.766 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ case "${pool}" in 2024-09-12T06:38:46.766 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 1 2024-09-12T06:38:46.766 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n '' ']' 2024-09-12T06:38:46.766 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ stress_write_image cluster2 mirror image_3 2024-09-12T06:38:46.766 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:38:46.766 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:38:46.766 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_3 2024-09-12T06:38:46.766 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T06:38:46.767 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local duration=11 2024-09-12T06:38:46.767 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set +e 2024-09-12T06:38:46.767 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_3 --debug-rbd=20 --debug-journaler=20 2024-09-12T06:38:50.265 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:100 IOs, 58040320 bytes 2024-09-12T06:38:53.190 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:200 IOs, 119474176 bytes 2024-09-12T06:38:55.553 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:300 IOs, 169260032 bytes 2024-09-12T06:38:57.616 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:400 IOs, 213867520 bytes 2024-09-12T06:38:57.782 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ error_code=124 2024-09-12T06:38:57.782 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set -e 2024-09-12T06:38:57.782 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T06:38:57.782 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:38:57.783 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:38:57.783 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_4 2024-09-12T06:38:57.783 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image_and_enable_mirror cluster2 mirror image_4 journal 128M 2024-09-12T06:38:57.783 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:38:57.783 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:38:57.783 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:38:57.783 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:38:57.783 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_4 2024-09-12T06:38:57.783 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:38:57.783 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local mode=journal 2024-09-12T06:38:57.783 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n journal ']' 2024-09-12T06:38:57.783 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:38:57.784 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image cluster2 mirror image_4 128M 2024-09-12T06:38:57.784 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:38:57.784 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:38:57.784 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:38:57.784 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:38:57.784 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_4 2024-09-12T06:38:57.784 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:38:57.784 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local size=128 2024-09-12T06:38:57.784 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n 128M ']' 2024-09-12T06:38:57.784 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ size=128M 2024-09-12T06:38:57.784 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:38:57.784 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_4 2024-09-12T06:38:57.869 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T06:38:57.870 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:38:57.870 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' pool = image ']' 2024-09-12T06:38:57.870 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ case "${pool}" in 2024-09-12T06:38:57.870 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 1 2024-09-12T06:38:57.870 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n '' ']' 2024-09-12T06:38:57.870 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ stress_write_image cluster2 mirror image_4 2024-09-12T06:38:57.870 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:38:57.870 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:38:57.870 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_4 2024-09-12T06:38:57.870 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T06:38:57.871 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local duration=8 2024-09-12T06:38:57.871 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set +e 2024-09-12T06:38:57.871 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_4 --debug-rbd=20 --debug-journaler=20 2024-09-12T06:38:58.023 INFO:tasks.ceph.cluster2.osd.0.smithi152.stderr:2024-09-12T06:38:58.024+0000 7fd61689f640 -1 reset not still connected to 0x55e76decd110 2024-09-12T06:38:59.782 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:100 IOs, 29997056 bytes 2024-09-12T06:39:01.999 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:200 IOs, 63751168 bytes 2024-09-12T06:39:04.887 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:300 IOs, 131605504 bytes 2024-09-12T06:39:05.882 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ error_code=124 2024-09-12T06:39:05.882 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set -e 2024-09-12T06:39:05.882 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T06:39:05.882 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:39:05.882 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:39:05.882 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_5 2024-09-12T06:39:05.882 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image_and_enable_mirror cluster2 mirror image_5 journal 128M 2024-09-12T06:39:05.882 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:39:05.883 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:39:05.883 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:39:05.883 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:39:05.883 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_5 2024-09-12T06:39:05.883 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:39:05.883 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local mode=journal 2024-09-12T06:39:05.883 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n journal ']' 2024-09-12T06:39:05.883 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:39:05.883 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image cluster2 mirror image_5 128M 2024-09-12T06:39:05.883 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:39:05.883 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:39:05.883 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:39:05.883 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:39:05.884 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_5 2024-09-12T06:39:05.884 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:39:05.884 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local size=128 2024-09-12T06:39:05.884 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n 128M ']' 2024-09-12T06:39:05.884 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ size=128M 2024-09-12T06:39:05.884 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:39:05.884 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_5 2024-09-12T06:39:05.944 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T06:39:05.944 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:39:05.944 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' pool = image ']' 2024-09-12T06:39:05.944 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ case "${pool}" in 2024-09-12T06:39:05.944 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 1 2024-09-12T06:39:05.944 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n '' ']' 2024-09-12T06:39:05.944 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ stress_write_image cluster2 mirror image_5 2024-09-12T06:39:05.945 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:39:05.945 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:39:05.945 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_5 2024-09-12T06:39:05.945 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T06:39:05.945 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local duration=11 2024-09-12T06:39:05.945 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set +e 2024-09-12T06:39:05.946 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_5 --debug-rbd=20 --debug-journaler=20 2024-09-12T06:39:06.169 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:100 IOs, 636928 bytes 2024-09-12T06:39:06.411 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:39:06.412+0000 7fac12a52640 -1 reset not still connected to 0x55749b1cbee0 2024-09-12T06:39:06.413 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:39:06.412+0000 7fac12a52640 -1 reset not still connected to 0x55749b2be0d0 2024-09-12T06:39:06.413 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:39:06.412+0000 7fac12a52640 -1 reset not still connected to 0x55749b2be8f0 2024-09-12T06:39:06.413 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:39:06.412+0000 7fac12a52640 -1 reset not still connected to 0x55749b2bed00 2024-09-12T06:39:06.413 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:39:06.413+0000 7fac12a52640 -1 reset not still connected to 0x55749b2bedd0 2024-09-12T06:39:06.413 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:39:06.413+0000 7fac12a52640 -1 reset not still connected to 0x55749b2bf6c0 2024-09-12T06:39:06.413 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:39:06.413+0000 7fac12a52640 -1 reset not still connected to 0x55749c9a7860 2024-09-12T06:39:09.369 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:200 IOs, 45046784 bytes 2024-09-12T06:39:12.867 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:300 IOs, 110608384 bytes 2024-09-12T06:39:14.065 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:400 IOs, 131254272 bytes 2024-09-12T06:39:15.817 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:39:15.819+0000 7fac12a52640 -1 reset not still connected to 0x55749b1ca5b0 2024-09-12T06:39:15.819 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:39:15.821+0000 7fac12a52640 -1 reset not still connected to 0x55749b1cbe10 2024-09-12T06:39:15.820 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:39:15.821+0000 7fac12a52640 -1 reset not still connected to 0x55749b1cbee0 2024-09-12T06:39:15.820 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:39:15.821+0000 7fac12a52640 -1 reset not still connected to 0x55749b2be0d0 2024-09-12T06:39:15.820 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:39:15.821+0000 7fac12a52640 -1 reset not still connected to 0x55749b2be8f0 2024-09-12T06:39:15.820 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:39:15.821+0000 7fac12a52640 -1 reset not still connected to 0x55749b2bed00 2024-09-12T06:39:15.820 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:39:15.821+0000 7fac12a52640 -1 reset not still connected to 0x55749b2bedd0 2024-09-12T06:39:15.820 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:39:15.821+0000 7fac12a52640 -1 reset not still connected to 0x55749b2bf6c0 2024-09-12T06:39:15.821 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:39:15.822+0000 7fac12a52640 -1 reset not still connected to 0x55749c9a7860 2024-09-12T06:39:15.821 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:39:15.822+0000 7fac12a52640 -1 reset not still connected to 0x55749e3c4750 2024-09-12T06:39:16.434 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:500 IOs, 194299904 bytes 2024-09-12T06:39:16.957 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ error_code=124 2024-09-12T06:39:16.957 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set -e 2024-09-12T06:39:16.957 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T06:39:16.958 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:39:16.958 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:39:16.958 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_6 2024-09-12T06:39:16.958 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image_and_enable_mirror cluster2 mirror image_6 journal 128M 2024-09-12T06:39:16.958 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:39:16.958 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:39:16.958 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:39:16.958 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:39:16.958 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_6 2024-09-12T06:39:16.958 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:39:16.958 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local mode=journal 2024-09-12T06:39:16.958 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n journal ']' 2024-09-12T06:39:16.958 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:39:16.958 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image cluster2 mirror image_6 128M 2024-09-12T06:39:16.959 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:39:16.959 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:39:16.959 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:39:16.959 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:39:16.959 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_6 2024-09-12T06:39:16.959 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:39:16.959 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local size=128 2024-09-12T06:39:16.959 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n 128M ']' 2024-09-12T06:39:16.959 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ size=128M 2024-09-12T06:39:16.959 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:39:16.959 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_6 2024-09-12T06:39:17.029 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T06:39:17.029 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:39:17.029 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' pool = image ']' 2024-09-12T06:39:17.030 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ case "${pool}" in 2024-09-12T06:39:17.030 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 1 2024-09-12T06:39:17.030 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n '' ']' 2024-09-12T06:39:17.030 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ stress_write_image cluster2 mirror image_6 2024-09-12T06:39:17.030 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:39:17.030 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:39:17.030 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_6 2024-09-12T06:39:17.030 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T06:39:17.031 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local duration=13 2024-09-12T06:39:17.031 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set +e 2024-09-12T06:39:17.031 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_6 --debug-rbd=20 --debug-journaler=20 2024-09-12T06:39:17.532 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:39:17.533+0000 7fa325ae1640 -1 reset not still connected to 0x55f85ba925b0 2024-09-12T06:39:18.425 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:100 IOs, 23497728 bytes 2024-09-12T06:39:20.526 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:200 IOs, 67348480 bytes 2024-09-12T06:39:22.168 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:300 IOs, 95729664 bytes 2024-09-12T06:39:24.520 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:400 IOs, 153148416 bytes 2024-09-12T06:39:28.053 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:500 IOs, 200032256 bytes 2024-09-12T06:39:30.039 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ error_code=124 2024-09-12T06:39:30.040 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set -e 2024-09-12T06:39:30.040 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T06:39:30.040 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:39:30.040 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:39:30.040 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_7 2024-09-12T06:39:30.040 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image_and_enable_mirror cluster2 mirror image_7 journal 128M 2024-09-12T06:39:30.040 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:39:30.040 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:39:30.040 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:39:30.040 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:39:30.040 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_7 2024-09-12T06:39:30.040 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:39:30.040 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local mode=journal 2024-09-12T06:39:30.041 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n journal ']' 2024-09-12T06:39:30.041 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:39:30.041 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image cluster2 mirror image_7 128M 2024-09-12T06:39:30.041 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:39:30.041 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:39:30.041 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:39:30.041 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:39:30.041 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_7 2024-09-12T06:39:30.041 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:39:30.041 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local size=128 2024-09-12T06:39:30.041 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n 128M ']' 2024-09-12T06:39:30.042 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ size=128M 2024-09-12T06:39:30.042 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:39:30.042 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_7 2024-09-12T06:39:30.103 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T06:39:30.103 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:39:30.103 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' pool = image ']' 2024-09-12T06:39:30.103 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ case "${pool}" in 2024-09-12T06:39:30.103 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 1 2024-09-12T06:39:30.104 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n '' ']' 2024-09-12T06:39:30.104 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ stress_write_image cluster2 mirror image_7 2024-09-12T06:39:30.104 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:39:30.104 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:39:30.104 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_7 2024-09-12T06:39:30.104 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T06:39:30.105 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local duration=7 2024-09-12T06:39:30.105 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set +e 2024-09-12T06:39:30.105 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_7 --debug-rbd=20 --debug-journaler=20 2024-09-12T06:39:36.021 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:100 IOs, 85477376 bytes 2024-09-12T06:39:37.118 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ error_code=124 2024-09-12T06:39:37.118 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set -e 2024-09-12T06:39:37.118 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T06:39:37.118 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:39:37.118 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:39:37.118 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_8 2024-09-12T06:39:37.118 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image_and_enable_mirror cluster2 mirror image_8 journal 128M 2024-09-12T06:39:37.119 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:39:37.119 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:39:37.119 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:39:37.119 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:39:37.119 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_8 2024-09-12T06:39:37.119 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:39:37.119 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local mode=journal 2024-09-12T06:39:37.119 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n journal ']' 2024-09-12T06:39:37.119 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:39:37.119 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image cluster2 mirror image_8 128M 2024-09-12T06:39:37.119 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:39:37.119 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:39:37.119 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:39:37.120 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:39:37.120 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_8 2024-09-12T06:39:37.120 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:39:37.120 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local size=128 2024-09-12T06:39:37.120 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n 128M ']' 2024-09-12T06:39:37.120 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ size=128M 2024-09-12T06:39:37.120 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:39:37.120 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_8 2024-09-12T06:39:37.183 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T06:39:37.184 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:39:37.184 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' pool = image ']' 2024-09-12T06:39:37.184 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ case "${pool}" in 2024-09-12T06:39:37.184 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 1 2024-09-12T06:39:37.184 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n '' ']' 2024-09-12T06:39:37.184 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ stress_write_image cluster2 mirror image_8 2024-09-12T06:39:37.185 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:39:37.185 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:39:37.185 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_8 2024-09-12T06:39:37.185 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T06:39:37.185 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local duration=14 2024-09-12T06:39:37.185 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set +e 2024-09-12T06:39:37.186 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_8 --debug-rbd=20 --debug-journaler=20 2024-09-12T06:39:39.742 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:100 IOs, 50322432 bytes 2024-09-12T06:39:41.619 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:200 IOs, 65953792 bytes 2024-09-12T06:39:44.925 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:300 IOs, 111472640 bytes 2024-09-12T06:39:46.998 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:39:46.999+0000 7fa325ae1640 -1 reset not still connected to 0x55f85a4fcdd0 2024-09-12T06:39:46.998 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:39:46.999+0000 7fa325ae1640 -1 reset not still connected to 0x55f8849a1040 2024-09-12T06:39:47.778 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:400 IOs, 171697152 bytes 2024-09-12T06:39:51.196 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ error_code=124 2024-09-12T06:39:51.197 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set -e 2024-09-12T06:39:51.197 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T06:39:51.197 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:39:51.197 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:39:51.197 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_9 2024-09-12T06:39:51.197 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image_and_enable_mirror cluster2 mirror image_9 journal 128M 2024-09-12T06:39:51.197 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:39:51.197 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:39:51.198 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:39:51.198 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:39:51.198 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_9 2024-09-12T06:39:51.198 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:39:51.198 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local mode=journal 2024-09-12T06:39:51.198 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n journal ']' 2024-09-12T06:39:51.198 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:39:51.198 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image cluster2 mirror image_9 128M 2024-09-12T06:39:51.198 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:39:51.198 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:39:51.198 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:39:51.198 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:39:51.198 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_9 2024-09-12T06:39:51.198 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:39:51.199 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local size=128 2024-09-12T06:39:51.199 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n 128M ']' 2024-09-12T06:39:51.199 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ size=128M 2024-09-12T06:39:51.199 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:39:51.199 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_9 2024-09-12T06:39:51.262 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T06:39:51.262 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:39:51.262 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' pool = image ']' 2024-09-12T06:39:51.262 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ case "${pool}" in 2024-09-12T06:39:51.262 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 1 2024-09-12T06:39:51.263 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n '' ']' 2024-09-12T06:39:51.263 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ stress_write_image cluster2 mirror image_9 2024-09-12T06:39:51.263 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:39:51.263 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:39:51.263 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_9 2024-09-12T06:39:51.263 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T06:39:51.263 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local duration=5 2024-09-12T06:39:51.264 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set +e 2024-09-12T06:39:51.264 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_9 --debug-rbd=20 --debug-journaler=20 2024-09-12T06:39:53.473 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:100 IOs, 35525632 bytes 2024-09-12T06:39:56.115 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:200 IOs, 70146048 bytes 2024-09-12T06:39:56.273 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ error_code=124 2024-09-12T06:39:56.273 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set -e 2024-09-12T06:39:56.273 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T06:39:56.273 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:39:56.273 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:39:56.273 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_10 2024-09-12T06:39:56.273 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image_and_enable_mirror cluster2 mirror image_10 journal 128M 2024-09-12T06:39:56.273 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:39:56.274 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:39:56.274 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:39:56.274 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:39:56.274 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_10 2024-09-12T06:39:56.274 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:39:56.274 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local mode=journal 2024-09-12T06:39:56.274 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n journal ']' 2024-09-12T06:39:56.274 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:39:56.274 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image cluster2 mirror image_10 128M 2024-09-12T06:39:56.274 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:39:56.274 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:39:56.274 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:39:56.275 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:39:56.275 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_10 2024-09-12T06:39:56.275 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:39:56.275 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local size=128 2024-09-12T06:39:56.275 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n 128M ']' 2024-09-12T06:39:56.275 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ size=128M 2024-09-12T06:39:56.275 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:39:56.275 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_10 2024-09-12T06:39:56.341 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T06:39:56.341 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:39:56.341 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' pool = image ']' 2024-09-12T06:39:56.341 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ case "${pool}" in 2024-09-12T06:39:56.341 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 1 2024-09-12T06:39:56.342 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n '' ']' 2024-09-12T06:39:56.342 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ stress_write_image cluster2 mirror image_10 2024-09-12T06:39:56.342 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:39:56.342 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:39:56.342 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_10 2024-09-12T06:39:56.342 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T06:39:56.343 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local duration=11 2024-09-12T06:39:56.343 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set +e 2024-09-12T06:39:56.343 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_10 --debug-rbd=20 --debug-journaler=20 2024-09-12T06:40:01.237 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:100 IOs, 74533888 bytes 2024-09-12T06:40:06.882 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:200 IOs, 198705152 bytes 2024-09-12T06:40:07.354 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ error_code=124 2024-09-12T06:40:07.354 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set -e 2024-09-12T06:40:07.354 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T06:40:07.354 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:40:07.354 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:40:07.354 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_11 2024-09-12T06:40:07.354 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image_and_enable_mirror cluster2 mirror image_11 journal 128M 2024-09-12T06:40:07.354 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:40:07.354 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:40:07.355 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:40:07.355 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:40:07.355 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_11 2024-09-12T06:40:07.355 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:40:07.355 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local mode=journal 2024-09-12T06:40:07.355 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n journal ']' 2024-09-12T06:40:07.355 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:40:07.355 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image cluster2 mirror image_11 128M 2024-09-12T06:40:07.355 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:40:07.355 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:40:07.355 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:40:07.355 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:40:07.355 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_11 2024-09-12T06:40:07.355 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:40:07.356 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local size=128 2024-09-12T06:40:07.356 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n 128M ']' 2024-09-12T06:40:07.356 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ size=128M 2024-09-12T06:40:07.356 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:40:07.356 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_11 2024-09-12T06:40:07.431 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T06:40:07.431 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:40:07.431 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' pool = image ']' 2024-09-12T06:40:07.431 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ case "${pool}" in 2024-09-12T06:40:07.431 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 1 2024-09-12T06:40:07.431 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n '' ']' 2024-09-12T06:40:07.431 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ stress_write_image cluster2 mirror image_11 2024-09-12T06:40:07.431 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:40:07.431 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:40:07.431 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_11 2024-09-12T06:40:07.431 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T06:40:07.432 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local duration=5 2024-09-12T06:40:07.432 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set +e 2024-09-12T06:40:07.432 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_11 --debug-rbd=20 --debug-journaler=20 2024-09-12T06:40:08.044 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:100 IOs, 6871040 bytes 2024-09-12T06:40:08.414 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:40:08.415+0000 7fa325ae1640 -1 reset not still connected to 0x55f85a304a90 2024-09-12T06:40:08.414 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:40:08.415+0000 7fa325ae1640 -1 reset not still connected to 0x55f85ac224e0 2024-09-12T06:40:08.414 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:40:08.415+0000 7fa325ae1640 -1 reset not still connected to 0x55f85ba925b0 2024-09-12T06:40:08.415 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:40:08.417+0000 7fa325ae1640 -1 reset not still connected to 0x55f85bac29c0 2024-09-12T06:40:09.180 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:40:09.181+0000 7f80d15bb640 -1 reset not still connected to 0x556b02ee8c30 2024-09-12T06:40:11.392 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:200 IOs, 51693568 bytes 2024-09-12T06:40:12.445 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ error_code=124 2024-09-12T06:40:12.445 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set -e 2024-09-12T06:40:12.445 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T06:40:12.446 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:40:12.446 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:40:12.446 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_12 2024-09-12T06:40:12.446 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image_and_enable_mirror cluster2 mirror image_12 journal 128M 2024-09-12T06:40:12.446 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:40:12.446 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:40:12.446 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:40:12.446 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:40:12.446 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_12 2024-09-12T06:40:12.446 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:40:12.446 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local mode=journal 2024-09-12T06:40:12.446 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n journal ']' 2024-09-12T06:40:12.446 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:40:12.447 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image cluster2 mirror image_12 128M 2024-09-12T06:40:12.447 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:40:12.447 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:40:12.447 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:40:12.447 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:40:12.447 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_12 2024-09-12T06:40:12.447 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:40:12.447 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local size=128 2024-09-12T06:40:12.447 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n 128M ']' 2024-09-12T06:40:12.447 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ size=128M 2024-09-12T06:40:12.448 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:40:12.448 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_12 2024-09-12T06:40:12.547 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T06:40:12.547 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:40:12.547 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' pool = image ']' 2024-09-12T06:40:12.547 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ case "${pool}" in 2024-09-12T06:40:12.547 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 1 2024-09-12T06:40:12.547 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n '' ']' 2024-09-12T06:40:12.547 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ stress_write_image cluster2 mirror image_12 2024-09-12T06:40:12.547 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:40:12.547 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:40:12.547 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_12 2024-09-12T06:40:12.547 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T06:40:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local duration=6 2024-09-12T06:40:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set +e 2024-09-12T06:40:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_12 --debug-rbd=20 --debug-journaler=20 2024-09-12T06:40:13.304 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:100 IOs, 8810496 bytes 2024-09-12T06:40:16.327 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:200 IOs, 39041024 bytes 2024-09-12T06:40:17.927 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:40:17.928+0000 7fa325ae1640 -1 reset not still connected to 0x55f859179380 2024-09-12T06:40:17.927 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:40:17.928+0000 7fa325ae1640 -1 reset not still connected to 0x55f85a304a90 2024-09-12T06:40:17.928 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:40:17.928+0000 7fa325ae1640 -1 reset not still connected to 0x55f85ac224e0 2024-09-12T06:40:17.928 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:40:17.928+0000 7fa325ae1640 -1 reset not still connected to 0x55f85ba925b0 2024-09-12T06:40:18.006 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:300 IOs, 70887424 bytes 2024-09-12T06:40:18.211 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:400 IOs, 71505920 bytes 2024-09-12T06:40:18.560 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ error_code=124 2024-09-12T06:40:18.561 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set -e 2024-09-12T06:40:18.561 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T06:40:18.561 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:40:18.561 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:40:18.561 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_13 2024-09-12T06:40:18.561 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image_and_enable_mirror cluster2 mirror image_13 journal 128M 2024-09-12T06:40:18.561 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:40:18.561 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:40:18.561 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:40:18.561 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:40:18.561 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_13 2024-09-12T06:40:18.561 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:40:18.562 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local mode=journal 2024-09-12T06:40:18.562 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n journal ']' 2024-09-12T06:40:18.562 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:40:18.562 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image cluster2 mirror image_13 128M 2024-09-12T06:40:18.562 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:40:18.562 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:40:18.562 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:40:18.562 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:40:18.562 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_13 2024-09-12T06:40:18.562 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:40:18.562 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local size=128 2024-09-12T06:40:18.562 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n 128M ']' 2024-09-12T06:40:18.562 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ size=128M 2024-09-12T06:40:18.562 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:40:18.562 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_13 2024-09-12T06:40:18.660 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T06:40:18.660 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:40:18.660 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' pool = image ']' 2024-09-12T06:40:18.660 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ case "${pool}" in 2024-09-12T06:40:18.660 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 1 2024-09-12T06:40:18.661 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n '' ']' 2024-09-12T06:40:18.661 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ stress_write_image cluster2 mirror image_13 2024-09-12T06:40:18.661 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:40:18.661 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:40:18.661 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_13 2024-09-12T06:40:18.661 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T06:40:18.661 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local duration=13 2024-09-12T06:40:18.662 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set +e 2024-09-12T06:40:18.662 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_13 --debug-rbd=20 --debug-journaler=20 2024-09-12T06:40:23.005 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:100 IOs, 84488192 bytes 2024-09-12T06:40:25.133 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:200 IOs, 131717120 bytes 2024-09-12T06:40:30.473 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:300 IOs, 206064640 bytes 2024-09-12T06:40:31.672 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ error_code=124 2024-09-12T06:40:31.672 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set -e 2024-09-12T06:40:31.672 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T06:40:31.672 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:40:31.672 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:40:31.672 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_14 2024-09-12T06:40:31.672 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image_and_enable_mirror cluster2 mirror image_14 journal 128M 2024-09-12T06:40:31.672 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:40:31.672 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:40:31.672 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:40:31.673 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:40:31.673 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_14 2024-09-12T06:40:31.673 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:40:31.673 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local mode=journal 2024-09-12T06:40:31.673 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n journal ']' 2024-09-12T06:40:31.673 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:40:31.673 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image cluster2 mirror image_14 128M 2024-09-12T06:40:31.673 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:40:31.673 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:40:31.673 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:40:31.673 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:40:31.673 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_14 2024-09-12T06:40:31.673 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:40:31.673 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local size=128 2024-09-12T06:40:31.674 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n 128M ']' 2024-09-12T06:40:31.674 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ size=128M 2024-09-12T06:40:31.674 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:40:31.674 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_14 2024-09-12T06:40:31.760 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T06:40:31.760 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:40:31.760 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' pool = image ']' 2024-09-12T06:40:31.761 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ case "${pool}" in 2024-09-12T06:40:31.761 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 1 2024-09-12T06:40:31.761 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n '' ']' 2024-09-12T06:40:31.761 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ stress_write_image cluster2 mirror image_14 2024-09-12T06:40:31.761 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:40:31.761 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:40:31.761 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_14 2024-09-12T06:40:31.761 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T06:40:31.762 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local duration=9 2024-09-12T06:40:31.763 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set +e 2024-09-12T06:40:31.763 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_14 --debug-rbd=20 --debug-journaler=20 2024-09-12T06:40:35.103 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:100 IOs, 38817792 bytes 2024-09-12T06:40:35.433 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:200 IOs, 40657920 bytes 2024-09-12T06:40:38.664 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:300 IOs, 95333376 bytes 2024-09-12T06:40:40.154 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:400 IOs, 119415808 bytes 2024-09-12T06:40:40.771 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ error_code=124 2024-09-12T06:40:40.801 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set -e 2024-09-12T06:40:40.801 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T06:40:40.801 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:40:40.801 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:40:40.801 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_15 2024-09-12T06:40:40.801 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image_and_enable_mirror cluster2 mirror image_15 journal 128M 2024-09-12T06:40:40.801 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:40:40.801 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:40:40.802 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:40:40.802 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:40:40.802 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_15 2024-09-12T06:40:40.802 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:40:40.802 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local mode=journal 2024-09-12T06:40:40.802 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n journal ']' 2024-09-12T06:40:40.802 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:40:40.802 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image cluster2 mirror image_15 128M 2024-09-12T06:40:40.802 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:40:40.802 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:40:40.802 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:40:40.802 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:40:40.802 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_15 2024-09-12T06:40:40.803 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:40:40.803 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local size=128 2024-09-12T06:40:40.803 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n 128M ']' 2024-09-12T06:40:40.803 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ size=128M 2024-09-12T06:40:40.803 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:40:40.803 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_15 2024-09-12T06:40:40.857 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T06:40:40.857 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:40:40.857 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' pool = image ']' 2024-09-12T06:40:40.857 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ case "${pool}" in 2024-09-12T06:40:40.857 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 1 2024-09-12T06:40:40.857 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n '' ']' 2024-09-12T06:40:40.857 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ stress_write_image cluster2 mirror image_15 2024-09-12T06:40:40.858 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:40:40.858 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:40:40.858 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_15 2024-09-12T06:40:40.858 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T06:40:40.859 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local duration=8 2024-09-12T06:40:40.859 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set +e 2024-09-12T06:40:40.859 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_15 --debug-rbd=20 --debug-journaler=20 2024-09-12T06:40:41.000 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:40:41.002+0000 7f80d15bb640 -1 reset not still connected to 0x556b021731e0 2024-09-12T06:40:41.001 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:40:41.002+0000 7f80d15bb640 -1 reset not still connected to 0x556b02ee8dd0 2024-09-12T06:40:41.001 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:40:41.002+0000 7f80d15bb640 -1 reset not still connected to 0x556b030a2c30 2024-09-12T06:40:41.001 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:40:41.002+0000 7f80d15bb640 -1 reset not still connected to 0x556b043adba0 2024-09-12T06:40:41.001 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:40:41.002+0000 7f80d15bb640 -1 reset not still connected to 0x556b0c2c6750 2024-09-12T06:40:43.104 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:100 IOs, 32679936 bytes 2024-09-12T06:40:43.301 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:40:43.302+0000 7fac12a52640 -1 reset not still connected to 0x55749c9a7860 2024-09-12T06:40:43.301 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:40:43.302+0000 7fac12a52640 -1 reset not still connected to 0x55749ca4d790 2024-09-12T06:40:43.301 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:40:43.302+0000 7fac12a52640 -1 reset not still connected to 0x55749e3c4750 2024-09-12T06:40:47.389 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:40:47.390+0000 7fac12a52640 -1 reset not still connected to 0x55749b2bf110 2024-09-12T06:40:47.390 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:40:47.391+0000 7fac12a52640 -1 reset not still connected to 0x55749b2bfee0 2024-09-12T06:40:47.390 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:40:47.391+0000 7fac12a52640 -1 reset not still connected to 0x55749c9e9380 2024-09-12T06:40:47.391 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:40:47.392+0000 7fac12a52640 -1 reset not still connected to 0x55749d402410 2024-09-12T06:40:47.391 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:40:47.392+0000 7fac12a52640 -1 reset not still connected to 0x55749d403ba0 2024-09-12T06:40:47.391 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:40:47.392+0000 7fac12a52640 -1 reset not still connected to 0x55749d7ca270 2024-09-12T06:40:47.391 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:40:47.392+0000 7fac12a52640 -1 reset not still connected to 0x5574a2fa7a00 2024-09-12T06:40:47.391 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:40:47.392+0000 7fac12a52640 -1 reset not still connected to 0x5574a62b6d00 2024-09-12T06:40:48.755 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:200 IOs, 137175040 bytes 2024-09-12T06:40:48.868 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ error_code=124 2024-09-12T06:40:48.869 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set -e 2024-09-12T06:40:48.869 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T06:40:48.869 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:40:48.869 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:40:48.869 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_16 2024-09-12T06:40:48.869 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image_and_enable_mirror cluster2 mirror image_16 journal 128M 2024-09-12T06:40:48.869 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:40:48.870 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:40:48.870 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:40:48.870 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:40:48.870 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_16 2024-09-12T06:40:48.870 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:40:48.870 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local mode=journal 2024-09-12T06:40:48.870 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n journal ']' 2024-09-12T06:40:48.870 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:40:48.870 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image cluster2 mirror image_16 128M 2024-09-12T06:40:48.870 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:40:48.870 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:40:48.870 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:40:48.871 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:40:48.871 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_16 2024-09-12T06:40:48.871 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:40:48.871 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local size=128 2024-09-12T06:40:48.871 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n 128M ']' 2024-09-12T06:40:48.871 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ size=128M 2024-09-12T06:40:48.871 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:40:48.871 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_16 2024-09-12T06:40:48.944 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T06:40:48.944 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:40:48.944 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' pool = image ']' 2024-09-12T06:40:48.945 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ case "${pool}" in 2024-09-12T06:40:48.945 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 1 2024-09-12T06:40:48.945 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n '' ']' 2024-09-12T06:40:48.945 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ stress_write_image cluster2 mirror image_16 2024-09-12T06:40:48.945 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:40:48.945 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:40:48.945 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_16 2024-09-12T06:40:48.945 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T06:40:48.946 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local duration=12 2024-09-12T06:40:48.946 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set +e 2024-09-12T06:40:48.946 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_16 --debug-rbd=20 --debug-journaler=20 2024-09-12T06:40:49.418 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:40:49.419+0000 7f80d15bb640 -1 reset not still connected to 0x556b02ee8c30 2024-09-12T06:40:49.821 INFO:tasks.ceph.cluster2.osd.2.smithi152.stderr:2024-09-12T06:40:49.821+0000 7f4789beb640 -1 reset not still connected to 0x56163df360d0 2024-09-12T06:40:50.015 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:100 IOs, 14145536 bytes 2024-09-12T06:40:52.517 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:200 IOs, 56175616 bytes 2024-09-12T06:40:52.679 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:40:52.680+0000 7f80d15bb640 -1 reset not still connected to 0x556b030a2c30 2024-09-12T06:40:52.679 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:40:52.680+0000 7f80d15bb640 -1 reset not still connected to 0x556b043adba0 2024-09-12T06:40:52.680 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:40:52.682+0000 7f80d15bb640 -1 reset not still connected to 0x556b0c2c6750 2024-09-12T06:40:52.682 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:40:52.683+0000 7f80d15bb640 -1 reset not still connected to 0x556b32cfef70 2024-09-12T06:40:53.024 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:40:53.026+0000 7fac12a52640 -1 reset not still connected to 0x55749b2bf6c0 2024-09-12T06:40:53.536 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:40:53.536+0000 7f80d15bb640 -1 reset not still connected to 0x556b030a2c30 2024-09-12T06:40:53.537 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:40:53.536+0000 7f80d15bb640 -1 reset not still connected to 0x556b043adba0 2024-09-12T06:40:53.539 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:40:53.539+0000 7f80d15bb640 -1 reset not still connected to 0x556b0c2c6750 2024-09-12T06:40:53.539 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:40:53.541+0000 7f80d15bb640 -1 reset not still connected to 0x556b32cfef70 2024-09-12T06:40:55.130 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:40:55.131+0000 7fa325ae1640 -1 reset not still connected to 0x55f85a4fcdd0 2024-09-12T06:40:55.130 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:40:55.131+0000 7fa325ae1640 -1 reset not still connected to 0x55f85cc67040 2024-09-12T06:40:55.134 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:40:55.136+0000 7fa325ae1640 -1 reset not still connected to 0x55f863611110 2024-09-12T06:40:55.134 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:40:55.136+0000 7fa325ae1640 -1 reset not still connected to 0x55f8849a1040 2024-09-12T06:40:55.134 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:40:55.136+0000 7fa325ae1640 -1 reset not still connected to 0x55f8849a11e0 2024-09-12T06:40:55.134 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:40:55.136+0000 7fa325ae1640 -1 reset not still connected to 0x55f88b2cd5f0 2024-09-12T06:40:55.169 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:40:55.169+0000 7fa325ae1640 -1 reset not still connected to 0x55f85a4fcd00 2024-09-12T06:40:55.169 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:40:55.169+0000 7fa325ae1640 -1 reset not still connected to 0x55f85a4fcdd0 2024-09-12T06:40:55.169 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:40:55.169+0000 7fa325ae1640 -1 reset not still connected to 0x55f85cc67040 2024-09-12T06:40:55.170 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:40:55.171+0000 7fa325ae1640 -1 reset not still connected to 0x55f863611110 2024-09-12T06:40:55.170 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:40:55.171+0000 7fa325ae1640 -1 reset not still connected to 0x55f8849a1040 2024-09-12T06:40:55.170 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:40:55.171+0000 7fa325ae1640 -1 reset not still connected to 0x55f8849a11e0 2024-09-12T06:40:55.170 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:40:55.172+0000 7fa325ae1640 -1 reset not still connected to 0x55f88b2cd5f0 2024-09-12T06:40:56.089 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:300 IOs, 103280640 bytes 2024-09-12T06:40:57.639 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:40:57.638+0000 7fa325ae1640 -1 reset not still connected to 0x55f859178c30 2024-09-12T06:40:57.640 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:40:57.642+0000 7fa325ae1640 -1 reset not still connected to 0x55f85a4fcd00 2024-09-12T06:40:57.641 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:40:57.642+0000 7fa325ae1640 -1 reset not still connected to 0x55f85a4fcdd0 2024-09-12T06:40:57.641 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:40:57.642+0000 7fa325ae1640 -1 reset not still connected to 0x55f85cc67040 2024-09-12T06:40:57.642 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:40:57.644+0000 7fa325ae1640 -1 reset not still connected to 0x55f863611110 2024-09-12T06:40:57.642 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:40:57.644+0000 7fa325ae1640 -1 reset not still connected to 0x55f8849a1040 2024-09-12T06:40:57.643 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:40:57.644+0000 7fa325ae1640 -1 reset not still connected to 0x55f8849a11e0 2024-09-12T06:40:57.643 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:40:57.644+0000 7fa325ae1640 -1 reset not still connected to 0x55f88b2cd5f0 2024-09-12T06:40:57.975 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:400 IOs, 134151168 bytes 2024-09-12T06:40:58.936 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:40:58.937+0000 7fac12a52640 -1 reset not still connected to 0x5574bdabf1e0 2024-09-12T06:41:00.955 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ error_code=124 2024-09-12T06:41:00.955 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set -e 2024-09-12T06:41:00.955 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T06:41:00.955 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:41:00.955 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:41:00.955 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_17 2024-09-12T06:41:00.955 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image_and_enable_mirror cluster2 mirror image_17 journal 128M 2024-09-12T06:41:00.956 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:41:00.956 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:41:00.956 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:41:00.956 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:41:00.956 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_17 2024-09-12T06:41:00.956 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:41:00.956 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local mode=journal 2024-09-12T06:41:00.956 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n journal ']' 2024-09-12T06:41:00.956 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:41:00.956 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image cluster2 mirror image_17 128M 2024-09-12T06:41:00.956 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:41:00.956 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:41:00.956 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:41:00.957 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:41:00.957 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_17 2024-09-12T06:41:00.957 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:41:00.957 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local size=128 2024-09-12T06:41:00.957 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n 128M ']' 2024-09-12T06:41:00.957 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ size=128M 2024-09-12T06:41:00.957 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:41:00.957 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_17 2024-09-12T06:41:01.029 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T06:41:01.029 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:41:01.029 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' pool = image ']' 2024-09-12T06:41:01.029 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ case "${pool}" in 2024-09-12T06:41:01.029 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 1 2024-09-12T06:41:01.030 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n '' ']' 2024-09-12T06:41:01.030 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ stress_write_image cluster2 mirror image_17 2024-09-12T06:41:01.030 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:41:01.030 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:41:01.030 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_17 2024-09-12T06:41:01.030 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T06:41:01.030 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local duration=5 2024-09-12T06:41:01.031 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set +e 2024-09-12T06:41:01.031 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_17 --debug-rbd=20 --debug-journaler=20 2024-09-12T06:41:02.208 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:100 IOs, 27990016 bytes 2024-09-12T06:41:04.991 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:41:04.992+0000 7fac12a52640 -1 reset not still connected to 0x55749b2bfee0 2024-09-12T06:41:04.992 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:41:04.992+0000 7fac12a52640 -1 reset not still connected to 0x55749c9e9380 2024-09-12T06:41:04.992 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:41:04.992+0000 7fac12a52640 -1 reset not still connected to 0x55749d402410 2024-09-12T06:41:04.992 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:41:04.992+0000 7fac12a52640 -1 reset not still connected to 0x55749d403ba0 2024-09-12T06:41:04.992 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:41:04.992+0000 7fac12a52640 -1 reset not still connected to 0x55749d7ca270 2024-09-12T06:41:04.992 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:41:04.992+0000 7fac12a52640 -1 reset not still connected to 0x5574a2fa7a00 2024-09-12T06:41:04.992 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:41:04.992+0000 7fac12a52640 -1 reset not still connected to 0x5574a62b6d00 2024-09-12T06:41:04.992 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:41:04.992+0000 7fac12a52640 -1 reset not still connected to 0x5574bdabf1e0 2024-09-12T06:41:05.885 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:200 IOs, 92355584 bytes 2024-09-12T06:41:06.039 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ error_code=124 2024-09-12T06:41:06.040 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set -e 2024-09-12T06:41:06.040 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T06:41:06.040 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:41:06.040 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:41:06.040 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_18 2024-09-12T06:41:06.040 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image_and_enable_mirror cluster2 mirror image_18 journal 128M 2024-09-12T06:41:06.040 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:41:06.040 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:41:06.040 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:41:06.041 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:41:06.041 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_18 2024-09-12T06:41:06.041 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:41:06.041 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local mode=journal 2024-09-12T06:41:06.041 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n journal ']' 2024-09-12T06:41:06.041 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:41:06.041 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image cluster2 mirror image_18 128M 2024-09-12T06:41:06.041 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:41:06.041 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:41:06.041 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:41:06.041 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:41:06.042 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_18 2024-09-12T06:41:06.042 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:41:06.042 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local size=128 2024-09-12T06:41:06.042 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n 128M ']' 2024-09-12T06:41:06.042 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ size=128M 2024-09-12T06:41:06.042 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:41:06.042 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_18 2024-09-12T06:41:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T06:41:06.109 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:41:06.109 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' pool = image ']' 2024-09-12T06:41:06.109 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ case "${pool}" in 2024-09-12T06:41:06.109 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 1 2024-09-12T06:41:06.109 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n '' ']' 2024-09-12T06:41:06.109 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ stress_write_image cluster2 mirror image_18 2024-09-12T06:41:06.113 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:41:06.113 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:41:06.113 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_18 2024-09-12T06:41:06.113 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T06:41:06.114 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local duration=5 2024-09-12T06:41:06.114 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set +e 2024-09-12T06:41:06.114 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_18 --debug-rbd=20 --debug-journaler=20 2024-09-12T06:41:08.105 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:100 IOs, 36252672 bytes 2024-09-12T06:41:10.162 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:200 IOs, 73580544 bytes 2024-09-12T06:41:11.121 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ error_code=124 2024-09-12T06:41:11.121 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set -e 2024-09-12T06:41:11.121 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T06:41:11.121 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:41:11.122 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:41:11.122 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_19 2024-09-12T06:41:11.122 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image_and_enable_mirror cluster2 mirror image_19 journal 128M 2024-09-12T06:41:11.122 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:41:11.122 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:41:11.122 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:41:11.122 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:41:11.122 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_19 2024-09-12T06:41:11.122 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:41:11.122 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local mode=journal 2024-09-12T06:41:11.122 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n journal ']' 2024-09-12T06:41:11.122 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:41:11.122 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image cluster2 mirror image_19 128M 2024-09-12T06:41:11.123 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:41:11.123 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:41:11.123 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:41:11.123 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:41:11.123 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_19 2024-09-12T06:41:11.123 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:41:11.123 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local size=128 2024-09-12T06:41:11.123 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n 128M ']' 2024-09-12T06:41:11.123 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ size=128M 2024-09-12T06:41:11.123 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:41:11.123 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_19 2024-09-12T06:41:11.201 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T06:41:11.201 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:41:11.202 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' pool = image ']' 2024-09-12T06:41:11.202 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ case "${pool}" in 2024-09-12T06:41:11.202 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 1 2024-09-12T06:41:11.202 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n '' ']' 2024-09-12T06:41:11.202 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ stress_write_image cluster2 mirror image_19 2024-09-12T06:41:11.202 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:41:11.202 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:41:11.202 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_19 2024-09-12T06:41:11.202 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T06:41:11.203 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local duration=5 2024-09-12T06:41:11.203 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set +e 2024-09-12T06:41:11.203 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_19 --debug-rbd=20 --debug-journaler=20 2024-09-12T06:41:11.297 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:41:11.298+0000 7fa325ae1640 -1 reset not still connected to 0x55f85a4fc9c0 2024-09-12T06:41:11.298 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:41:11.298+0000 7fa325ae1640 -1 reset not still connected to 0x55f85a4fcdd0 2024-09-12T06:41:11.298 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:41:11.298+0000 7fa325ae1640 -1 reset not still connected to 0x55f85bac3520 2024-09-12T06:41:11.298 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:41:11.298+0000 7fa325ae1640 -1 reset not still connected to 0x55f85cc67040 2024-09-12T06:41:11.298 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:41:11.298+0000 7fa325ae1640 -1 reset not still connected to 0x55f863611110 2024-09-12T06:41:11.298 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:41:11.298+0000 7fa325ae1640 -1 reset not still connected to 0x55f8849a11e0 2024-09-12T06:41:11.298 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:41:11.298+0000 7fa325ae1640 -1 reset not still connected to 0x55f88b2cd5f0 2024-09-12T06:41:14.515 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:100 IOs, 52313088 bytes 2024-09-12T06:41:14.954 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:200 IOs, 80520192 bytes 2024-09-12T06:41:16.212 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ error_code=124 2024-09-12T06:41:16.212 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set -e 2024-09-12T06:41:16.212 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T06:41:16.212 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:41:16.212 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:41:16.212 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_20 2024-09-12T06:41:16.212 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image_and_enable_mirror cluster2 mirror image_20 journal 128M 2024-09-12T06:41:16.212 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:41:16.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:41:16.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:41:16.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:41:16.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_20 2024-09-12T06:41:16.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:41:16.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local mode=journal 2024-09-12T06:41:16.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n journal ']' 2024-09-12T06:41:16.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:41:16.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image cluster2 mirror image_20 128M 2024-09-12T06:41:16.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:41:16.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:41:16.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:41:16.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:41:16.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_20 2024-09-12T06:41:16.214 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:41:16.214 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local size=128 2024-09-12T06:41:16.214 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n 128M ']' 2024-09-12T06:41:16.214 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ size=128M 2024-09-12T06:41:16.214 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:41:16.214 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_20 2024-09-12T06:41:16.278 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T06:41:16.278 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:41:16.278 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' pool = image ']' 2024-09-12T06:41:16.278 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ case "${pool}" in 2024-09-12T06:41:16.278 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 1 2024-09-12T06:41:16.278 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n '' ']' 2024-09-12T06:41:16.278 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ stress_write_image cluster2 mirror image_20 2024-09-12T06:41:16.278 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:41:16.278 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:41:16.278 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_20 2024-09-12T06:41:16.278 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T06:41:16.279 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local duration=10 2024-09-12T06:41:16.279 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set +e 2024-09-12T06:41:16.279 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_20 --debug-rbd=20 --debug-journaler=20 2024-09-12T06:41:16.616 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:41:16.616+0000 7fac12a52640 -1 reset not still connected to 0x55749b1cbe10 2024-09-12T06:41:16.616 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:41:16.616+0000 7fac12a52640 -1 reset not still connected to 0x55749b1cbee0 2024-09-12T06:41:16.616 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:41:16.616+0000 7fac12a52640 -1 reset not still connected to 0x55749b2be0d0 2024-09-12T06:41:16.616 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:41:16.616+0000 7fac12a52640 -1 reset not still connected to 0x55749b2be8f0 2024-09-12T06:41:16.616 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:41:16.616+0000 7fac12a52640 -1 reset not still connected to 0x55749b2bed00 2024-09-12T06:41:16.616 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:41:16.616+0000 7fac12a52640 -1 reset not still connected to 0x55749b2bedd0 2024-09-12T06:41:16.616 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:41:16.616+0000 7fac12a52640 -1 reset not still connected to 0x55749b2bf6c0 2024-09-12T06:41:16.616 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:41:16.617+0000 7fac12a52640 -1 reset not still connected to 0x55749c9a7860 2024-09-12T06:41:16.616 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:41:16.617+0000 7fac12a52640 -1 reset not still connected to 0x55749ca4d790 2024-09-12T06:41:16.617 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:41:16.618+0000 7fac12a52640 -1 reset not still connected to 0x55749d3c4750 2024-09-12T06:41:16.617 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:41:16.618+0000 7fac12a52640 -1 reset not still connected to 0x55749e3c4750 2024-09-12T06:41:17.900 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:100 IOs, 24581120 bytes 2024-09-12T06:41:19.506 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:200 IOs, 42601472 bytes 2024-09-12T06:41:21.588 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:41:21.588+0000 7f80d15bb640 -1 reset not still connected to 0x556b32cfef70 2024-09-12T06:41:22.362 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:300 IOs, 107660288 bytes 2024-09-12T06:41:22.641 INFO:tasks.ceph.cluster2.osd.1.smithi152.stderr:2024-09-12T06:41:22.642+0000 7f33e805b640 -1 bdev(0x55c73b90ec00 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 1 2024-09-12T06:41:22.644 INFO:tasks.ceph.cluster2.osd.1.smithi152.stderr:2024-09-12T06:41:22.646+0000 7f33e6858640 -1 bdev(0x55c73b90ec00 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 3 2024-09-12T06:41:22.969 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:41:22.971+0000 7fa325ae1640 -1 reset not still connected to 0x55f8636984e0 2024-09-12T06:41:22.970 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:41:22.971+0000 7fa325ae1640 -1 reset not still connected to 0x55f863699040 2024-09-12T06:41:23.622 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:400 IOs, 145862656 bytes 2024-09-12T06:41:25.710 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:500 IOs, 187995136 bytes 2024-09-12T06:41:26.287 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ error_code=124 2024-09-12T06:41:26.287 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set -e 2024-09-12T06:41:26.287 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T06:41:26.288 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:41:26.288 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:41:26.288 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_21 2024-09-12T06:41:26.288 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image_and_enable_mirror cluster2 mirror image_21 journal 128M 2024-09-12T06:41:26.288 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:41:26.288 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:41:26.288 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:41:26.288 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:41:26.288 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_21 2024-09-12T06:41:26.288 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:41:26.288 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local mode=journal 2024-09-12T06:41:26.289 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n journal ']' 2024-09-12T06:41:26.289 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:41:26.289 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image cluster2 mirror image_21 128M 2024-09-12T06:41:26.289 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:41:26.289 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:41:26.289 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:41:26.289 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:41:26.289 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_21 2024-09-12T06:41:26.289 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:41:26.289 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local size=128 2024-09-12T06:41:26.289 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n 128M ']' 2024-09-12T06:41:26.289 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ size=128M 2024-09-12T06:41:26.289 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:41:26.290 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_21 2024-09-12T06:41:26.352 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T06:41:26.352 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:41:26.352 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' pool = image ']' 2024-09-12T06:41:26.352 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ case "${pool}" in 2024-09-12T06:41:26.352 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 1 2024-09-12T06:41:26.352 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n '' ']' 2024-09-12T06:41:26.352 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ stress_write_image cluster2 mirror image_21 2024-09-12T06:41:26.352 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:41:26.352 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:41:26.352 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_21 2024-09-12T06:41:26.353 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T06:41:26.353 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local duration=11 2024-09-12T06:41:26.353 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set +e 2024-09-12T06:41:26.353 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_21 --debug-rbd=20 --debug-journaler=20 2024-09-12T06:41:26.727 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:100 IOs, 3844096 bytes 2024-09-12T06:41:28.872 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:41:28.873+0000 7fa325ae1640 -1 reset not still connected to 0x55f880b4e4e0 2024-09-12T06:41:28.872 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:41:28.873+0000 7fa325ae1640 -1 reset not still connected to 0x55f8849a1040 2024-09-12T06:41:28.872 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:41:28.873+0000 7fa325ae1640 -1 reset not still connected to 0x55f8849a11e0 2024-09-12T06:41:28.872 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:41:28.873+0000 7fa325ae1640 -1 reset not still connected to 0x55f88b2cd5f0 2024-09-12T06:41:28.940 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:41:28.941+0000 7fa325ae1640 -1 reset not still connected to 0x55f85cc67040 2024-09-12T06:41:28.941 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:41:28.941+0000 7fa325ae1640 -1 reset not still connected to 0x55f863611110 2024-09-12T06:41:28.941 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:41:28.942+0000 7fa325ae1640 -1 reset not still connected to 0x55f880b4e4e0 2024-09-12T06:41:28.944 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:41:28.946+0000 7fa325ae1640 -1 reset not still connected to 0x55f8849a1040 2024-09-12T06:41:28.944 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:41:28.946+0000 7fa325ae1640 -1 reset not still connected to 0x55f8849a11e0 2024-09-12T06:41:28.944 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:41:28.946+0000 7fa325ae1640 -1 reset not still connected to 0x55f88b2cd5f0 2024-09-12T06:41:29.606 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:200 IOs, 30844928 bytes 2024-09-12T06:41:31.979 INFO:tasks.ceph.cluster2.osd.1.smithi152.stderr:2024-09-12T06:41:31.980+0000 7f33f9c90640 -1 reset not still connected to 0x55c758d9fd40 2024-09-12T06:41:33.145 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:41:33.146+0000 7fac12a52640 -1 reset not still connected to 0x55749b2bfee0 2024-09-12T06:41:33.146 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:41:33.147+0000 7fac12a52640 -1 reset not still connected to 0x55749d402410 2024-09-12T06:41:33.146 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:41:33.147+0000 7fac12a52640 -1 reset not still connected to 0x55749d403ba0 2024-09-12T06:41:33.146 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:41:33.147+0000 7fac12a52640 -1 reset not still connected to 0x55749d7ca270 2024-09-12T06:41:33.146 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:41:33.147+0000 7fac12a52640 -1 reset not still connected to 0x5574a2fa7a00 2024-09-12T06:41:33.149 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:41:33.148+0000 7fac12a52640 -1 reset not still connected to 0x5574a62b6d00 2024-09-12T06:41:33.149 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:41:33.148+0000 7fac12a52640 -1 reset not still connected to 0x5574bdabf1e0 2024-09-12T06:41:33.562 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:300 IOs, 91806720 bytes 2024-09-12T06:41:35.821 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:41:35.821+0000 7fa325ae1640 -1 reset not still connected to 0x55f85a4fcd00 2024-09-12T06:41:35.822 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:41:35.823+0000 7fa325ae1640 -1 reset not still connected to 0x55f85a4fcdd0 2024-09-12T06:41:35.822 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:41:35.823+0000 7fa325ae1640 -1 reset not still connected to 0x55f85bac3520 2024-09-12T06:41:35.822 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:41:35.823+0000 7fa325ae1640 -1 reset not still connected to 0x55f85cc67040 2024-09-12T06:41:35.822 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:41:35.823+0000 7fa325ae1640 -1 reset not still connected to 0x55f863611110 2024-09-12T06:41:35.822 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:41:35.823+0000 7fa325ae1640 -1 reset not still connected to 0x55f880b4e4e0 2024-09-12T06:41:35.822 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:41:35.823+0000 7fa325ae1640 -1 reset not still connected to 0x55f8849a1040 2024-09-12T06:41:35.822 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:41:35.823+0000 7fa325ae1640 -1 reset not still connected to 0x55f8849a11e0 2024-09-12T06:41:35.822 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:41:35.823+0000 7fa325ae1640 -1 reset not still connected to 0x55f88b2cd5f0 2024-09-12T06:41:37.362 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ error_code=124 2024-09-12T06:41:37.362 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set -e 2024-09-12T06:41:37.362 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T06:41:37.362 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:41:37.362 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:41:37.362 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_22 2024-09-12T06:41:37.362 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image_and_enable_mirror cluster2 mirror image_22 journal 128M 2024-09-12T06:41:37.362 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:41:37.362 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:41:37.362 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:41:37.363 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:41:37.363 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_22 2024-09-12T06:41:37.363 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:41:37.363 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local mode=journal 2024-09-12T06:41:37.363 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n journal ']' 2024-09-12T06:41:37.363 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:41:37.363 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image cluster2 mirror image_22 128M 2024-09-12T06:41:37.363 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:41:37.363 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:41:37.363 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:41:37.363 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:41:37.363 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_22 2024-09-12T06:41:37.363 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:41:37.364 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local size=128 2024-09-12T06:41:37.364 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n 128M ']' 2024-09-12T06:41:37.364 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ size=128M 2024-09-12T06:41:37.364 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:41:37.364 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_22 2024-09-12T06:41:37.451 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T06:41:37.451 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:41:37.451 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' pool = image ']' 2024-09-12T06:41:37.452 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ case "${pool}" in 2024-09-12T06:41:37.452 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 1 2024-09-12T06:41:37.452 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n '' ']' 2024-09-12T06:41:37.452 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ stress_write_image cluster2 mirror image_22 2024-09-12T06:41:37.452 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:41:37.452 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:41:37.452 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_22 2024-09-12T06:41:37.452 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T06:41:37.453 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local duration=7 2024-09-12T06:41:37.453 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set +e 2024-09-12T06:41:37.453 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_22 --debug-rbd=20 --debug-journaler=20 2024-09-12T06:41:43.925 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:100 IOs, 63371264 bytes 2024-09-12T06:41:44.462 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ error_code=124 2024-09-12T06:41:44.462 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set -e 2024-09-12T06:41:44.462 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T06:41:44.463 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:41:44.463 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:41:44.463 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_23 2024-09-12T06:41:44.463 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image_and_enable_mirror cluster2 mirror image_23 journal 128M 2024-09-12T06:41:44.463 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:41:44.464 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:41:44.464 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:41:44.464 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:41:44.464 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_23 2024-09-12T06:41:44.464 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:41:44.464 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local mode=journal 2024-09-12T06:41:44.465 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n journal ']' 2024-09-12T06:41:44.465 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:41:44.465 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image cluster2 mirror image_23 128M 2024-09-12T06:41:44.465 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:41:44.465 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:41:44.466 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:41:44.466 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:41:44.466 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_23 2024-09-12T06:41:44.466 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:41:44.466 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local size=128 2024-09-12T06:41:44.467 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n 128M ']' 2024-09-12T06:41:44.467 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ size=128M 2024-09-12T06:41:44.467 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:41:44.467 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_23 2024-09-12T06:41:44.553 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T06:41:44.554 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:41:44.554 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' pool = image ']' 2024-09-12T06:41:44.554 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ case "${pool}" in 2024-09-12T06:41:44.554 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 1 2024-09-12T06:41:44.554 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n '' ']' 2024-09-12T06:41:44.554 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ stress_write_image cluster2 mirror image_23 2024-09-12T06:41:44.555 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:41:44.555 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:41:44.555 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_23 2024-09-12T06:41:44.555 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T06:41:44.556 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local duration=5 2024-09-12T06:41:44.556 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set +e 2024-09-12T06:41:44.556 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_23 --debug-rbd=20 --debug-journaler=20 2024-09-12T06:41:49.476 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:41:49.477+0000 7f80d15bb640 -1 reset not still connected to 0x556b030a2c30 2024-09-12T06:41:49.476 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:41:49.477+0000 7f80d15bb640 -1 reset not still connected to 0x556b043adba0 2024-09-12T06:41:49.481 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:41:49.482+0000 7f80d15bb640 -1 reset not still connected to 0x556b14db6a90 2024-09-12T06:41:49.481 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:41:49.482+0000 7f80d15bb640 -1 reset not still connected to 0x556b32cfef70 2024-09-12T06:41:49.504 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:100 IOs, 82843648 bytes 2024-09-12T06:41:49.569 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ error_code=124 2024-09-12T06:41:49.569 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set -e 2024-09-12T06:41:49.569 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T06:41:49.569 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:41:49.569 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:41:49.569 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_24 2024-09-12T06:41:49.569 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image_and_enable_mirror cluster2 mirror image_24 journal 128M 2024-09-12T06:41:49.569 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:41:49.569 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:41:49.570 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:41:49.570 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:41:49.570 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_24 2024-09-12T06:41:49.570 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:41:49.570 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local mode=journal 2024-09-12T06:41:49.570 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n journal ']' 2024-09-12T06:41:49.570 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:41:49.570 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image cluster2 mirror image_24 128M 2024-09-12T06:41:49.570 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:41:49.570 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:41:49.570 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:41:49.570 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:41:49.570 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_24 2024-09-12T06:41:49.570 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:41:49.570 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local size=128 2024-09-12T06:41:49.571 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n 128M ']' 2024-09-12T06:41:49.571 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ size=128M 2024-09-12T06:41:49.571 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:41:49.571 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_24 2024-09-12T06:41:49.665 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T06:41:49.665 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:41:49.665 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' pool = image ']' 2024-09-12T06:41:49.666 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ case "${pool}" in 2024-09-12T06:41:49.666 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 1 2024-09-12T06:41:49.666 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n '' ']' 2024-09-12T06:41:49.666 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ stress_write_image cluster2 mirror image_24 2024-09-12T06:41:49.666 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:41:49.666 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:41:49.666 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_24 2024-09-12T06:41:49.667 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T06:41:49.667 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local duration=10 2024-09-12T06:41:49.667 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set +e 2024-09-12T06:41:49.668 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_24 --debug-rbd=20 --debug-journaler=20 2024-09-12T06:41:51.376 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:100 IOs, 22615040 bytes 2024-09-12T06:41:53.674 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:200 IOs, 47143936 bytes 2024-09-12T06:41:57.864 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:300 IOs, 134310912 bytes 2024-09-12T06:41:59.429 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:41:59.429+0000 7fa325ae1640 -1 reset not still connected to 0x55f85a4fc9c0 2024-09-12T06:41:59.429 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:41:59.429+0000 7fa325ae1640 -1 reset not still connected to 0x55f85a4fcd00 2024-09-12T06:41:59.429 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:41:59.429+0000 7fa325ae1640 -1 reset not still connected to 0x55f85a4fcdd0 2024-09-12T06:41:59.429 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:41:59.429+0000 7fa325ae1640 -1 reset not still connected to 0x55f85bac3520 2024-09-12T06:41:59.429 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:41:59.429+0000 7fa325ae1640 -1 reset not still connected to 0x55f85cc67040 2024-09-12T06:41:59.429 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:41:59.429+0000 7fa325ae1640 -1 reset not still connected to 0x55f863611110 2024-09-12T06:41:59.429 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:41:59.429+0000 7fa325ae1640 -1 reset not still connected to 0x55f880b4e4e0 2024-09-12T06:41:59.429 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:41:59.429+0000 7fa325ae1640 -1 reset not still connected to 0x55f8849a1040 2024-09-12T06:41:59.429 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:41:59.429+0000 7fa325ae1640 -1 reset not still connected to 0x55f8849a11e0 2024-09-12T06:41:59.429 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:41:59.429+0000 7fa325ae1640 -1 reset not still connected to 0x55f88b2cd5f0 2024-09-12T06:41:59.677 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ error_code=124 2024-09-12T06:41:59.677 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set -e 2024-09-12T06:41:59.677 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T06:41:59.677 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:41:59.677 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:41:59.677 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_25 2024-09-12T06:41:59.677 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image_and_enable_mirror cluster2 mirror image_25 journal 128M 2024-09-12T06:41:59.678 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:41:59.678 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:41:59.678 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:41:59.678 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:41:59.678 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_25 2024-09-12T06:41:59.678 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:41:59.678 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local mode=journal 2024-09-12T06:41:59.678 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n journal ']' 2024-09-12T06:41:59.678 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:41:59.678 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image cluster2 mirror image_25 128M 2024-09-12T06:41:59.678 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:41:59.678 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:41:59.678 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:41:59.679 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:41:59.679 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_25 2024-09-12T06:41:59.679 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:41:59.679 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local size=128 2024-09-12T06:41:59.679 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n 128M ']' 2024-09-12T06:41:59.679 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ size=128M 2024-09-12T06:41:59.679 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:41:59.679 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_25 2024-09-12T06:41:59.710 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:41:59.711+0000 7fa325ae1640 -1 reset not still connected to 0x55f85bac2b60 2024-09-12T06:41:59.711 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:41:59.712+0000 7fa325ae1640 -1 reset not still connected to 0x55f87faff040 2024-09-12T06:41:59.753 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T06:41:59.753 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:41:59.753 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' pool = image ']' 2024-09-12T06:41:59.753 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ case "${pool}" in 2024-09-12T06:41:59.753 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 1 2024-09-12T06:41:59.754 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n '' ']' 2024-09-12T06:41:59.754 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ stress_write_image cluster2 mirror image_25 2024-09-12T06:41:59.754 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:41:59.754 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:41:59.754 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_25 2024-09-12T06:41:59.754 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T06:41:59.755 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local duration=11 2024-09-12T06:41:59.755 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set +e 2024-09-12T06:41:59.755 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_25 --debug-rbd=20 --debug-journaler=20 2024-09-12T06:42:04.039 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:100 IOs, 58281984 bytes 2024-09-12T06:42:05.510 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:200 IOs, 78680064 bytes 2024-09-12T06:42:05.555 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:05.556+0000 7fa325ae1640 -1 reset not still connected to 0x55f863699040 2024-09-12T06:42:05.556 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:05.556+0000 7fa325ae1640 -1 reset not still connected to 0x55f87faff040 2024-09-12T06:42:07.438 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:300 IOs, 122870784 bytes 2024-09-12T06:42:09.127 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:400 IOs, 160056320 bytes 2024-09-12T06:42:09.992 INFO:tasks.ceph.cluster2.osd.1.smithi152.stderr:2024-09-12T06:42:09.988+0000 7f33f9c90640 -1 reset not still connected to 0x55c73c373a00 2024-09-12T06:42:09.992 INFO:tasks.ceph.cluster2.osd.1.smithi152.stderr:2024-09-12T06:42:09.988+0000 7f33f9c90640 -1 reset not still connected to 0x55c73c373ba0 2024-09-12T06:42:10.766 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ error_code=124 2024-09-12T06:42:10.766 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set -e 2024-09-12T06:42:10.766 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T06:42:10.766 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:42:10.767 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:42:10.767 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_26 2024-09-12T06:42:10.767 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image_and_enable_mirror cluster2 mirror image_26 journal 128M 2024-09-12T06:42:10.767 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:42:10.767 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:42:10.767 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:42:10.767 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:42:10.767 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_26 2024-09-12T06:42:10.767 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:42:10.767 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local mode=journal 2024-09-12T06:42:10.767 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n journal ']' 2024-09-12T06:42:10.767 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:42:10.767 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image cluster2 mirror image_26 128M 2024-09-12T06:42:10.767 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:42:10.767 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:42:10.768 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:42:10.768 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:42:10.768 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_26 2024-09-12T06:42:10.768 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:42:10.768 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local size=128 2024-09-12T06:42:10.768 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n 128M ']' 2024-09-12T06:42:10.768 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ size=128M 2024-09-12T06:42:10.768 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:42:10.768 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_26 2024-09-12T06:42:10.835 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T06:42:10.835 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:42:10.835 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' pool = image ']' 2024-09-12T06:42:10.835 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ case "${pool}" in 2024-09-12T06:42:10.836 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 1 2024-09-12T06:42:10.836 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n '' ']' 2024-09-12T06:42:10.836 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ stress_write_image cluster2 mirror image_26 2024-09-12T06:42:10.836 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:42:10.836 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:42:10.836 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_26 2024-09-12T06:42:10.836 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T06:42:10.837 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local duration=11 2024-09-12T06:42:10.837 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set +e 2024-09-12T06:42:10.837 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_26 --debug-rbd=20 --debug-journaler=20 2024-09-12T06:42:15.301 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:100 IOs, 70119424 bytes 2024-09-12T06:42:16.456 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:16.458+0000 7fa325ae1640 -1 reset not still connected to 0x55f85bac3520 2024-09-12T06:42:16.456 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:16.458+0000 7fa325ae1640 -1 reset not still connected to 0x55f85cc67040 2024-09-12T06:42:16.459 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:16.460+0000 7fa325ae1640 -1 reset not still connected to 0x55f863611110 2024-09-12T06:42:16.462 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:16.463+0000 7fa325ae1640 -1 reset not still connected to 0x55f88041e270 2024-09-12T06:42:16.462 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:16.463+0000 7fa325ae1640 -1 reset not still connected to 0x55f880b4e4e0 2024-09-12T06:42:16.462 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:16.463+0000 7fa325ae1640 -1 reset not still connected to 0x55f8849a1040 2024-09-12T06:42:16.462 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:16.463+0000 7fa325ae1640 -1 reset not still connected to 0x55f8849a11e0 2024-09-12T06:42:16.462 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:16.463+0000 7fa325ae1640 -1 reset not still connected to 0x55f8863b5790 2024-09-12T06:42:16.462 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:16.463+0000 7fa325ae1640 -1 reset not still connected to 0x55f88b2cd5f0 2024-09-12T06:42:16.725 INFO:tasks.ceph.cluster2.osd.1.smithi152.stderr:2024-09-12T06:42:16.727+0000 7f33ec864640 -1 bdev(0x55c73b90ec00 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 4 2024-09-12T06:42:16.727 INFO:tasks.ceph.cluster2.osd.1.smithi152.stderr:2024-09-12T06:42:16.728+0000 7f33e785a640 -1 bdev(0x55c73b90ec00 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 4 2024-09-12T06:42:17.066 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:200 IOs, 97773568 bytes 2024-09-12T06:42:18.541 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:42:18.542+0000 7fac12a52640 -1 reset not still connected to 0x55749b1cbe10 2024-09-12T06:42:18.541 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:42:18.542+0000 7fac12a52640 -1 reset not still connected to 0x55749b1cbee0 2024-09-12T06:42:18.541 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:42:18.542+0000 7fac12a52640 -1 reset not still connected to 0x55749b2be0d0 2024-09-12T06:42:18.542 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:42:18.542+0000 7fac12a52640 -1 reset not still connected to 0x55749b2be8f0 2024-09-12T06:42:18.542 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:42:18.542+0000 7fac12a52640 -1 reset not still connected to 0x55749b2bedd0 2024-09-12T06:42:18.542 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:42:18.542+0000 7fac12a52640 -1 reset not still connected to 0x55749b2bf6c0 2024-09-12T06:42:18.542 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:42:18.542+0000 7fac12a52640 -1 reset not still connected to 0x55749b2bfc70 2024-09-12T06:42:18.542 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:42:18.542+0000 7fac12a52640 -1 reset not still connected to 0x55749c9a7860 2024-09-12T06:42:18.542 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:42:18.542+0000 7fac12a52640 -1 reset not still connected to 0x55749ca4d790 2024-09-12T06:42:18.545 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:42:18.546+0000 7fac12a52640 -1 reset not still connected to 0x55749e3c4750 2024-09-12T06:42:18.545 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:42:18.546+0000 7fac12a52640 -1 reset not still connected to 0x55749e3c48f0 2024-09-12T06:42:20.514 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:20.516+0000 7fa325ae1640 -1 reset not still connected to 0x55f85bac2b60 2024-09-12T06:42:20.516 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:20.517+0000 7fa325ae1640 -1 reset not still connected to 0x55f86ee2e340 2024-09-12T06:42:20.516 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:20.517+0000 7fa325ae1640 -1 reset not still connected to 0x55f87faff040 2024-09-12T06:42:20.736 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:300 IOs, 146308096 bytes 2024-09-12T06:42:20.757 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:42:20.759+0000 7fac12a52640 -1 reset not still connected to 0x5574a2fa7a00 2024-09-12T06:42:20.764 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:42:20.766+0000 7fac12a52640 -1 reset not still connected to 0x5574a62b6d00 2024-09-12T06:42:20.765 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:42:20.766+0000 7fac12a52640 -1 reset not still connected to 0x5574bdabf110 2024-09-12T06:42:20.765 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:42:20.766+0000 7fac12a52640 -1 reset not still connected to 0x5574bdabf1e0 2024-09-12T06:42:21.847 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ error_code=124 2024-09-12T06:42:21.848 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set -e 2024-09-12T06:42:21.848 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T06:42:21.848 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:42:21.848 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:42:21.848 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_27 2024-09-12T06:42:21.848 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image_and_enable_mirror cluster2 mirror image_27 journal 128M 2024-09-12T06:42:21.848 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:42:21.848 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:42:21.848 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:42:21.848 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:42:21.848 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_27 2024-09-12T06:42:21.849 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:42:21.849 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local mode=journal 2024-09-12T06:42:21.849 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n journal ']' 2024-09-12T06:42:21.849 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:42:21.849 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image cluster2 mirror image_27 128M 2024-09-12T06:42:21.849 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:42:21.849 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:42:21.849 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:42:21.849 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:42:21.849 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_27 2024-09-12T06:42:21.849 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:42:21.849 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local size=128 2024-09-12T06:42:21.850 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n 128M ']' 2024-09-12T06:42:21.850 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ size=128M 2024-09-12T06:42:21.850 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:42:21.850 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_27 2024-09-12T06:42:21.951 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T06:42:21.952 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:42:21.952 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' pool = image ']' 2024-09-12T06:42:21.952 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ case "${pool}" in 2024-09-12T06:42:21.952 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 1 2024-09-12T06:42:21.952 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n '' ']' 2024-09-12T06:42:21.952 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ stress_write_image cluster2 mirror image_27 2024-09-12T06:42:21.952 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:42:21.952 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:42:21.952 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_27 2024-09-12T06:42:21.952 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T06:42:21.953 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local duration=8 2024-09-12T06:42:21.953 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set +e 2024-09-12T06:42:21.953 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_27 --debug-rbd=20 --debug-journaler=20 2024-09-12T06:42:25.038 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:100 IOs, 62315520 bytes 2024-09-12T06:42:26.966 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:26.963+0000 7fa325ae1640 -1 reset not still connected to 0x55f85ac229c0 2024-09-12T06:42:26.966 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:26.965+0000 7fa325ae1640 -1 reset not still connected to 0x55f85ba925b0 2024-09-12T06:42:26.966 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:26.967+0000 7fa325ae1640 -1 reset not still connected to 0x55f85bac2b60 2024-09-12T06:42:26.967 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:26.967+0000 7fa325ae1640 -1 reset not still connected to 0x55f8636984e0 2024-09-12T06:42:26.967 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:26.967+0000 7fa325ae1640 -1 reset not still connected to 0x55f863699040 2024-09-12T06:42:26.967 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:26.967+0000 7fa325ae1640 -1 reset not still connected to 0x55f86ee2e340 2024-09-12T06:42:26.967 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:26.967+0000 7fa325ae1640 -1 reset not still connected to 0x55f87faff040 2024-09-12T06:42:27.001 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:200 IOs, 74477568 bytes 2024-09-12T06:42:27.729 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:27.730+0000 7fa325ae1640 -1 reset not still connected to 0x55f8863b5790 2024-09-12T06:42:27.731 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:27.733+0000 7fa325ae1640 -1 reset not still connected to 0x55f88b2cd5f0 2024-09-12T06:42:28.905 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:42:28.906+0000 7fac12a52640 -1 reset not still connected to 0x55749b1cbee0 2024-09-12T06:42:28.905 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:42:28.907+0000 7fac12a52640 -1 reset not still connected to 0x55749b2be0d0 2024-09-12T06:42:28.907 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:42:28.907+0000 7fac12a52640 -1 reset not still connected to 0x55749b2be8f0 2024-09-12T06:42:28.907 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:42:28.909+0000 7fac12a52640 -1 reset not still connected to 0x55749b2bedd0 2024-09-12T06:42:28.908 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:42:28.910+0000 7fac12a52640 -1 reset not still connected to 0x55749b2bf6c0 2024-09-12T06:42:28.908 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:42:28.910+0000 7fac12a52640 -1 reset not still connected to 0x55749b2bfc70 2024-09-12T06:42:28.908 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:42:28.910+0000 7fac12a52640 -1 reset not still connected to 0x55749c9a7860 2024-09-12T06:42:28.911 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:42:28.911+0000 7fac12a52640 -1 reset not still connected to 0x55749ca4d790 2024-09-12T06:42:28.911 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:42:28.911+0000 7fac12a52640 -1 reset not still connected to 0x55749e3c4750 2024-09-12T06:42:28.911 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:42:28.911+0000 7fac12a52640 -1 reset not still connected to 0x55749e3c48f0 2024-09-12T06:42:29.962 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ error_code=124 2024-09-12T06:42:29.963 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set -e 2024-09-12T06:42:29.963 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T06:42:29.963 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:42:29.963 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:42:29.963 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_28 2024-09-12T06:42:29.963 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image_and_enable_mirror cluster2 mirror image_28 journal 128M 2024-09-12T06:42:29.963 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:42:29.963 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:42:29.963 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:42:29.963 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:42:29.963 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_28 2024-09-12T06:42:29.963 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:42:29.963 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local mode=journal 2024-09-12T06:42:29.964 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n journal ']' 2024-09-12T06:42:29.964 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:42:29.964 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image cluster2 mirror image_28 128M 2024-09-12T06:42:29.964 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:42:29.964 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:42:29.964 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:42:29.964 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:42:29.964 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_28 2024-09-12T06:42:29.964 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:42:29.964 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local size=128 2024-09-12T06:42:29.964 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n 128M ']' 2024-09-12T06:42:29.964 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ size=128M 2024-09-12T06:42:29.964 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:42:29.965 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_28 2024-09-12T06:42:30.047 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T06:42:30.047 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:42:30.047 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' pool = image ']' 2024-09-12T06:42:30.047 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ case "${pool}" in 2024-09-12T06:42:30.047 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 1 2024-09-12T06:42:30.047 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n '' ']' 2024-09-12T06:42:30.047 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ stress_write_image cluster2 mirror image_28 2024-09-12T06:42:30.047 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:42:30.047 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:42:30.048 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_28 2024-09-12T06:42:30.048 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T06:42:30.048 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local duration=5 2024-09-12T06:42:30.048 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set +e 2024-09-12T06:42:30.048 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_28 --debug-rbd=20 --debug-journaler=20 2024-09-12T06:42:31.882 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:100 IOs, 30389248 bytes 2024-09-12T06:42:32.564 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:32.565+0000 7fa325ae1640 -1 reset not still connected to 0x55f8636984e0 2024-09-12T06:42:32.564 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:32.565+0000 7fa325ae1640 -1 reset not still connected to 0x55f863699040 2024-09-12T06:42:32.564 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:32.565+0000 7fa325ae1640 -1 reset not still connected to 0x55f86ee2e340 2024-09-12T06:42:32.568 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:32.570+0000 7fa325ae1640 -1 reset not still connected to 0x55f87faff040 2024-09-12T06:42:35.057 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ error_code=124 2024-09-12T06:42:35.057 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set -e 2024-09-12T06:42:35.057 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T06:42:35.057 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:42:35.057 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:42:35.057 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_29 2024-09-12T06:42:35.057 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image_and_enable_mirror cluster2 mirror image_29 journal 128M 2024-09-12T06:42:35.057 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:42:35.057 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:42:35.057 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:42:35.058 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:42:35.058 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_29 2024-09-12T06:42:35.058 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:42:35.058 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local mode=journal 2024-09-12T06:42:35.058 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n journal ']' 2024-09-12T06:42:35.058 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:42:35.058 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image cluster2 mirror image_29 128M 2024-09-12T06:42:35.058 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:42:35.058 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:42:35.058 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:42:35.058 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:42:35.058 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_29 2024-09-12T06:42:35.058 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:42:35.059 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local size=128 2024-09-12T06:42:35.059 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n 128M ']' 2024-09-12T06:42:35.059 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ size=128M 2024-09-12T06:42:35.059 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:42:35.059 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_29 2024-09-12T06:42:35.158 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T06:42:35.158 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:42:35.158 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' pool = image ']' 2024-09-12T06:42:35.158 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ case "${pool}" in 2024-09-12T06:42:35.158 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 1 2024-09-12T06:42:35.158 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n '' ']' 2024-09-12T06:42:35.158 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ stress_write_image cluster2 mirror image_29 2024-09-12T06:42:35.158 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:42:35.159 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:42:35.159 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_29 2024-09-12T06:42:35.159 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T06:42:35.159 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local duration=7 2024-09-12T06:42:35.159 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set +e 2024-09-12T06:42:35.160 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_29 --debug-rbd=20 --debug-journaler=20 2024-09-12T06:42:37.617 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:37.618+0000 7fa325ae1640 -1 reset not still connected to 0x55f85a4fcd00 2024-09-12T06:42:37.619 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:37.620+0000 7fa325ae1640 -1 reset not still connected to 0x55f85a4fcdd0 2024-09-12T06:42:37.619 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:37.620+0000 7fa325ae1640 -1 reset not still connected to 0x55f85bac3520 2024-09-12T06:42:37.621 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:37.622+0000 7fa325ae1640 -1 reset not still connected to 0x55f85cc67040 2024-09-12T06:42:37.621 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:37.622+0000 7fa325ae1640 -1 reset not still connected to 0x55f863611110 2024-09-12T06:42:37.621 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:37.622+0000 7fa325ae1640 -1 reset not still connected to 0x55f86ee2fe10 2024-09-12T06:42:37.621 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:37.622+0000 7fa325ae1640 -1 reset not still connected to 0x55f88041e270 2024-09-12T06:42:37.621 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:37.622+0000 7fa325ae1640 -1 reset not still connected to 0x55f880b4e4e0 2024-09-12T06:42:37.621 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:37.622+0000 7fa325ae1640 -1 reset not still connected to 0x55f8849a1040 2024-09-12T06:42:37.621 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:37.622+0000 7fa325ae1640 -1 reset not still connected to 0x55f8849a11e0 2024-09-12T06:42:37.621 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:37.622+0000 7fa325ae1640 -1 reset not still connected to 0x55f8863b5790 2024-09-12T06:42:37.624 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:37.626+0000 7fa325ae1640 -1 reset not still connected to 0x55f88b2cd5f0 2024-09-12T06:42:38.933 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:38.935+0000 7fa325ae1640 -1 reset not still connected to 0x55f85ba925b0 2024-09-12T06:42:38.934 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:38.935+0000 7fa325ae1640 -1 reset not still connected to 0x55f85bac29c0 2024-09-12T06:42:38.934 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:38.935+0000 7fa325ae1640 -1 reset not still connected to 0x55f85bac2b60 2024-09-12T06:42:38.934 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:38.935+0000 7fa325ae1640 -1 reset not still connected to 0x55f85d61d380 2024-09-12T06:42:38.934 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:38.935+0000 7fa325ae1640 -1 reset not still connected to 0x55f8636984e0 2024-09-12T06:42:38.934 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:38.935+0000 7fa325ae1640 -1 reset not still connected to 0x55f863699040 2024-09-12T06:42:38.935 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:38.935+0000 7fa325ae1640 -1 reset not still connected to 0x55f86ee2e340 2024-09-12T06:42:38.935 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:38.935+0000 7fa325ae1640 -1 reset not still connected to 0x55f87faff040 2024-09-12T06:42:39.076 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:100 IOs, 44273664 bytes 2024-09-12T06:42:41.577 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:200 IOs, 84744192 bytes 2024-09-12T06:42:42.168 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ error_code=124 2024-09-12T06:42:42.168 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set -e 2024-09-12T06:42:42.169 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T06:42:42.169 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:42:42.169 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:42:42.169 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_30 2024-09-12T06:42:42.169 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image_and_enable_mirror cluster2 mirror image_30 journal 128M 2024-09-12T06:42:42.169 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:42:42.169 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:42:42.169 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:42:42.169 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:42:42.169 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_30 2024-09-12T06:42:42.169 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:42:42.169 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local mode=journal 2024-09-12T06:42:42.169 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n journal ']' 2024-09-12T06:42:42.169 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:42:42.170 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image cluster2 mirror image_30 128M 2024-09-12T06:42:42.170 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:42:42.170 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:42:42.170 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:42:42.170 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:42:42.170 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_30 2024-09-12T06:42:42.170 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:42:42.170 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local size=128 2024-09-12T06:42:42.170 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n 128M ']' 2024-09-12T06:42:42.170 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ size=128M 2024-09-12T06:42:42.170 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:42:42.170 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_30 2024-09-12T06:42:42.189 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:42.190+0000 7fa325ae1640 -1 reset not still connected to 0x55f859179380 2024-09-12T06:42:42.190 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:42.190+0000 7fa325ae1640 -1 reset not still connected to 0x55f85a304a90 2024-09-12T06:42:42.191 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:42.190+0000 7fa325ae1640 -1 reset not still connected to 0x55f85ba925b0 2024-09-12T06:42:42.191 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:42.190+0000 7fa325ae1640 -1 reset not still connected to 0x55f85bac29c0 2024-09-12T06:42:42.191 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:42.190+0000 7fa325ae1640 -1 reset not still connected to 0x55f85bac2b60 2024-09-12T06:42:42.191 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:42.190+0000 7fa325ae1640 -1 reset not still connected to 0x55f85d61d380 2024-09-12T06:42:42.191 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:42.190+0000 7fa325ae1640 -1 reset not still connected to 0x55f8636984e0 2024-09-12T06:42:42.191 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:42.190+0000 7fa325ae1640 -1 reset not still connected to 0x55f863699040 2024-09-12T06:42:42.191 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:42.190+0000 7fa325ae1640 -1 reset not still connected to 0x55f86ee2e340 2024-09-12T06:42:42.191 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:42.190+0000 7fa325ae1640 -1 reset not still connected to 0x55f87faff040 2024-09-12T06:42:42.241 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T06:42:42.241 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:42:42.241 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' pool = image ']' 2024-09-12T06:42:42.241 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ case "${pool}" in 2024-09-12T06:42:42.241 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 1 2024-09-12T06:42:42.241 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n '' ']' 2024-09-12T06:42:42.241 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ stress_write_image cluster2 mirror image_30 2024-09-12T06:42:42.241 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:42:42.241 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:42:42.242 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_30 2024-09-12T06:42:42.242 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T06:42:42.242 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local duration=5 2024-09-12T06:42:42.242 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set +e 2024-09-12T06:42:42.242 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_30 --debug-rbd=20 --debug-journaler=20 2024-09-12T06:42:43.574 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:100 IOs, 23027712 bytes 2024-09-12T06:42:44.034 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:44.034+0000 7fa325ae1640 -1 reset not still connected to 0x55f88b2cd5f0 2024-09-12T06:42:44.095 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:42:44.096+0000 7f80d15bb640 -1 reset not still connected to 0x556b02ee8c30 2024-09-12T06:42:44.095 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:42:44.096+0000 7f80d15bb640 -1 reset not still connected to 0x556b03866f70 2024-09-12T06:42:44.095 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:42:44.096+0000 7f80d15bb640 -1 reset not still connected to 0x556b33ff76c0 2024-09-12T06:42:44.613 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:44.614+0000 7fa325ae1640 -1 reset not still connected to 0x55f85ba925b0 2024-09-12T06:42:44.613 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:44.614+0000 7fa325ae1640 -1 reset not still connected to 0x55f85bac29c0 2024-09-12T06:42:44.613 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:44.614+0000 7fa325ae1640 -1 reset not still connected to 0x55f85bac2b60 2024-09-12T06:42:44.613 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:44.614+0000 7fa325ae1640 -1 reset not still connected to 0x55f85d61d380 2024-09-12T06:42:44.614 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:44.616+0000 7fa325ae1640 -1 reset not still connected to 0x55f863699040 2024-09-12T06:42:44.614 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:44.616+0000 7fa325ae1640 -1 reset not still connected to 0x55f86ee2e340 2024-09-12T06:42:44.615 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:44.616+0000 7fa325ae1640 -1 reset not still connected to 0x55f87faff040 2024-09-12T06:42:45.008 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:200 IOs, 56628224 bytes 2024-09-12T06:42:47.253 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ error_code=124 2024-09-12T06:42:47.253 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set -e 2024-09-12T06:42:47.253 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T06:42:47.253 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:42:47.253 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:42:47.253 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_31 2024-09-12T06:42:47.253 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image_and_enable_mirror cluster2 mirror image_31 journal 128M 2024-09-12T06:42:47.253 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:42:47.253 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:42:47.254 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:42:47.254 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:42:47.254 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_31 2024-09-12T06:42:47.254 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:42:47.254 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local mode=journal 2024-09-12T06:42:47.254 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n journal ']' 2024-09-12T06:42:47.254 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:42:47.254 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image cluster2 mirror image_31 128M 2024-09-12T06:42:47.254 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:42:47.254 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:42:47.254 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:42:47.254 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:42:47.255 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_31 2024-09-12T06:42:47.255 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:42:47.255 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local size=128 2024-09-12T06:42:47.255 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n 128M ']' 2024-09-12T06:42:47.255 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ size=128M 2024-09-12T06:42:47.255 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:42:47.255 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_31 2024-09-12T06:42:47.346 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T06:42:47.346 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:42:47.347 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' pool = image ']' 2024-09-12T06:42:47.347 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ case "${pool}" in 2024-09-12T06:42:47.347 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 1 2024-09-12T06:42:47.347 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n '' ']' 2024-09-12T06:42:47.347 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ stress_write_image cluster2 mirror image_31 2024-09-12T06:42:47.347 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:42:47.347 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:42:47.347 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_31 2024-09-12T06:42:47.347 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T06:42:47.348 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local duration=9 2024-09-12T06:42:47.348 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set +e 2024-09-12T06:42:47.348 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_31 --debug-rbd=20 --debug-journaler=20 2024-09-12T06:42:49.005 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:42:49.004+0000 7fac12a52640 -1 reset not still connected to 0x5574a2fa7a00 2024-09-12T06:42:49.005 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:42:49.004+0000 7fac12a52640 -1 reset not still connected to 0x5574a62b6d00 2024-09-12T06:42:49.005 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:42:49.007+0000 7fac12a52640 -1 reset not still connected to 0x5574bdabf1e0 2024-09-12T06:42:49.007 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:42:49.007+0000 7fac12a52640 -1 reset not still connected to 0x5574d5072c30 2024-09-12T06:42:50.699 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:42:50.695+0000 7fac12a52640 -1 reset not still connected to 0x55749b1cbee0 2024-09-12T06:42:50.699 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:42:50.695+0000 7fac12a52640 -1 reset not still connected to 0x55749b2be0d0 2024-09-12T06:42:50.700 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:42:50.695+0000 7fac12a52640 -1 reset not still connected to 0x55749b2be8f0 2024-09-12T06:42:50.700 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:42:50.695+0000 7fac12a52640 -1 reset not still connected to 0x55749b2bed00 2024-09-12T06:42:50.700 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:42:50.695+0000 7fac12a52640 -1 reset not still connected to 0x55749b2bedd0 2024-09-12T06:42:50.700 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:42:50.699+0000 7fac12a52640 -1 reset not still connected to 0x55749b2bf6c0 2024-09-12T06:42:50.702 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:42:50.699+0000 7fac12a52640 -1 reset not still connected to 0x55749b2bfc70 2024-09-12T06:42:50.703 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:42:50.699+0000 7fac12a52640 -1 reset not still connected to 0x55749c9a7860 2024-09-12T06:42:50.703 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:42:50.699+0000 7fac12a52640 -1 reset not still connected to 0x55749ca4d790 2024-09-12T06:42:50.703 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:42:50.701+0000 7fac12a52640 -1 reset not still connected to 0x55749d3c4750 2024-09-12T06:42:50.703 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:42:50.701+0000 7fac12a52640 -1 reset not still connected to 0x55749e3c4750 2024-09-12T06:42:50.703 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:42:50.701+0000 7fac12a52640 -1 reset not still connected to 0x55749e3c48f0 2024-09-12T06:42:50.703 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:42:50.701+0000 7fac12a52640 -1 reset not still connected to 0x55749e3c4d00 2024-09-12T06:42:50.703 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:42:50.703+0000 7fac12a52640 -1 reset not still connected to 0x5574b7fd3c70 2024-09-12T06:42:50.887 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:100 IOs, 57052160 bytes 2024-09-12T06:42:52.392 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:200 IOs, 57663488 bytes 2024-09-12T06:42:52.723 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:52.722+0000 7fa325ae1640 -1 reset not still connected to 0x55f85ba925b0 2024-09-12T06:42:52.723 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:52.722+0000 7fa325ae1640 -1 reset not still connected to 0x55f85bac29c0 2024-09-12T06:42:52.723 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:52.722+0000 7fa325ae1640 -1 reset not still connected to 0x55f85bac2b60 2024-09-12T06:42:52.723 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:52.722+0000 7fa325ae1640 -1 reset not still connected to 0x55f85cc67930 2024-09-12T06:42:52.723 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:52.722+0000 7fa325ae1640 -1 reset not still connected to 0x55f85d61d380 2024-09-12T06:42:52.723 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:52.722+0000 7fa325ae1640 -1 reset not still connected to 0x55f8636984e0 2024-09-12T06:42:52.723 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:52.722+0000 7fa325ae1640 -1 reset not still connected to 0x55f863699040 2024-09-12T06:42:52.723 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:52.722+0000 7fa325ae1640 -1 reset not still connected to 0x55f86ee2e340 2024-09-12T06:42:52.724 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:52.724+0000 7fa325ae1640 -1 reset not still connected to 0x55f87faff040 2024-09-12T06:42:53.964 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:53.965+0000 7fa325ae1640 -1 reset not still connected to 0x55f859178ea0 2024-09-12T06:42:53.967 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:53.968+0000 7fa325ae1640 -1 reset not still connected to 0x55f859179380 2024-09-12T06:42:53.967 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:53.968+0000 7fa325ae1640 -1 reset not still connected to 0x55f85a304a90 2024-09-12T06:42:53.970 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:53.971+0000 7fa325ae1640 -1 reset not still connected to 0x55f85ac224e0 2024-09-12T06:42:53.972 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:53.973+0000 7fa325ae1640 -1 reset not still connected to 0x55f85ac229c0 2024-09-12T06:42:53.972 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:53.973+0000 7fa325ae1640 -1 reset not still connected to 0x55f85ba925b0 2024-09-12T06:42:53.972 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:53.973+0000 7fa325ae1640 -1 reset not still connected to 0x55f85bac29c0 2024-09-12T06:42:53.972 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:53.973+0000 7fa325ae1640 -1 reset not still connected to 0x55f85bac2b60 2024-09-12T06:42:53.972 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:53.973+0000 7fa325ae1640 -1 reset not still connected to 0x55f85cc67930 2024-09-12T06:42:53.972 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:53.973+0000 7fa325ae1640 -1 reset not still connected to 0x55f85d61d380 2024-09-12T06:42:53.972 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:53.973+0000 7fa325ae1640 -1 reset not still connected to 0x55f8636984e0 2024-09-12T06:42:53.972 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:53.973+0000 7fa325ae1640 -1 reset not still connected to 0x55f863699040 2024-09-12T06:42:53.973 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:53.973+0000 7fa325ae1640 -1 reset not still connected to 0x55f86ee2e340 2024-09-12T06:42:53.973 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:53.973+0000 7fa325ae1640 -1 reset not still connected to 0x55f87faff040 2024-09-12T06:42:55.301 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:300 IOs, 103833600 bytes 2024-09-12T06:42:55.794 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:400 IOs, 110662656 bytes 2024-09-12T06:42:56.192 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:500 IOs, 114600960 bytes 2024-09-12T06:42:56.359 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ error_code=124 2024-09-12T06:42:56.359 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set -e 2024-09-12T06:42:56.359 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T06:42:56.359 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:42:56.359 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:42:56.359 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_32 2024-09-12T06:42:56.359 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image_and_enable_mirror cluster2 mirror image_32 journal 128M 2024-09-12T06:42:56.359 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:42:56.359 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:42:56.359 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:42:56.359 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:42:56.360 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_32 2024-09-12T06:42:56.360 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:42:56.360 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local mode=journal 2024-09-12T06:42:56.360 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n journal ']' 2024-09-12T06:42:56.360 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:42:56.360 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image cluster2 mirror image_32 128M 2024-09-12T06:42:56.360 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:42:56.360 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:42:56.360 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:42:56.360 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:42:56.360 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_32 2024-09-12T06:42:56.360 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:42:56.360 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local size=128 2024-09-12T06:42:56.360 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n 128M ']' 2024-09-12T06:42:56.360 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ size=128M 2024-09-12T06:42:56.361 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:42:56.361 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_32 2024-09-12T06:42:56.425 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T06:42:56.425 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:42:56.426 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' pool = image ']' 2024-09-12T06:42:56.426 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ case "${pool}" in 2024-09-12T06:42:56.426 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 1 2024-09-12T06:42:56.426 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n '' ']' 2024-09-12T06:42:56.426 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ stress_write_image cluster2 mirror image_32 2024-09-12T06:42:56.426 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:42:56.426 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:42:56.426 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_32 2024-09-12T06:42:56.426 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T06:42:56.427 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local duration=7 2024-09-12T06:42:56.427 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set +e 2024-09-12T06:42:56.427 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_32 --debug-rbd=20 --debug-journaler=20 2024-09-12T06:42:58.890 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:58.886+0000 7fa325ae1640 -1 reset not still connected to 0x55f859178c30 2024-09-12T06:42:58.892 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:58.892+0000 7fa325ae1640 -1 reset not still connected to 0x55f85a2509c0 2024-09-12T06:42:58.892 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:58.892+0000 7fa325ae1640 -1 reset not still connected to 0x55f85a4fc9c0 2024-09-12T06:42:58.892 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:58.892+0000 7fa325ae1640 -1 reset not still connected to 0x55f85a4fcd00 2024-09-12T06:42:58.892 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:58.892+0000 7fa325ae1640 -1 reset not still connected to 0x55f85cc67040 2024-09-12T06:42:58.892 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:58.892+0000 7fa325ae1640 -1 reset not still connected to 0x55f86ee2fe10 2024-09-12T06:42:58.893 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:58.892+0000 7fa325ae1640 -1 reset not still connected to 0x55f88041e270 2024-09-12T06:42:58.893 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:58.892+0000 7fa325ae1640 -1 reset not still connected to 0x55f884309c70 2024-09-12T06:42:58.893 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:58.892+0000 7fa325ae1640 -1 reset not still connected to 0x55f8849a1040 2024-09-12T06:42:58.893 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:58.892+0000 7fa325ae1640 -1 reset not still connected to 0x55f8849a11e0 2024-09-12T06:42:58.893 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:42:58.892+0000 7fa325ae1640 -1 reset not still connected to 0x55f88b2cd5f0 2024-09-12T06:42:58.977 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:100 IOs, 37330944 bytes 2024-09-12T06:43:02.265 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:200 IOs, 95245312 bytes 2024-09-12T06:43:02.859 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:300 IOs, 100487168 bytes 2024-09-12T06:43:03.438 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ error_code=124 2024-09-12T06:43:03.438 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set -e 2024-09-12T06:43:03.439 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T06:43:03.439 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:43:03.439 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:43:03.439 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_33 2024-09-12T06:43:03.439 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image_and_enable_mirror cluster2 mirror image_33 journal 128M 2024-09-12T06:43:03.439 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:43:03.439 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:43:03.439 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:43:03.439 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:43:03.439 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_33 2024-09-12T06:43:03.439 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:43:03.439 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local mode=journal 2024-09-12T06:43:03.439 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n journal ']' 2024-09-12T06:43:03.439 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:43:03.439 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image cluster2 mirror image_33 128M 2024-09-12T06:43:03.440 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:43:03.440 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:43:03.440 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:43:03.440 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:43:03.440 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_33 2024-09-12T06:43:03.440 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:43:03.440 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local size=128 2024-09-12T06:43:03.440 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n 128M ']' 2024-09-12T06:43:03.440 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ size=128M 2024-09-12T06:43:03.441 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:43:03.441 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_33 2024-09-12T06:43:03.520 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T06:43:03.521 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:43:03.521 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' pool = image ']' 2024-09-12T06:43:03.521 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ case "${pool}" in 2024-09-12T06:43:03.521 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 1 2024-09-12T06:43:03.521 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n '' ']' 2024-09-12T06:43:03.521 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ stress_write_image cluster2 mirror image_33 2024-09-12T06:43:03.521 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:43:03.521 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:43:03.521 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_33 2024-09-12T06:43:03.521 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T06:43:03.522 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local duration=7 2024-09-12T06:43:03.522 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set +e 2024-09-12T06:43:03.522 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_33 --debug-rbd=20 --debug-journaler=20 2024-09-12T06:43:05.071 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:43:05.069+0000 7fac12a52640 -1 reset not still connected to 0x55749b2bfee0 2024-09-12T06:43:05.071 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:43:05.069+0000 7fac12a52640 -1 reset not still connected to 0x55749c9e9380 2024-09-12T06:43:05.071 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:43:05.069+0000 7fac12a52640 -1 reset not still connected to 0x55749d403ba0 2024-09-12T06:43:05.071 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:43:05.069+0000 7fac12a52640 -1 reset not still connected to 0x55749d7ca270 2024-09-12T06:43:05.074 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:43:05.074+0000 7fac12a52640 -1 reset not still connected to 0x55749d7ca8f0 2024-09-12T06:43:05.074 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:43:05.074+0000 7fac12a52640 -1 reset not still connected to 0x55749e05cf70 2024-09-12T06:43:05.074 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:43:05.074+0000 7fac12a52640 -1 reset not still connected to 0x5574a2fa7a00 2024-09-12T06:43:05.074 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:43:05.074+0000 7fac12a52640 -1 reset not still connected to 0x5574a62b6d00 2024-09-12T06:43:05.078 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:43:05.079+0000 7fac12a52640 -1 reset not still connected to 0x5574bdabf1e0 2024-09-12T06:43:05.078 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:43:05.079+0000 7fac12a52640 -1 reset not still connected to 0x5574d1d6da00 2024-09-12T06:43:05.078 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:43:05.079+0000 7fac12a52640 -1 reset not still connected to 0x5574d5072c30 2024-09-12T06:43:07.890 INFO:tasks.ceph.cluster2.osd.2.smithi152.stderr:2024-09-12T06:43:07.890+0000 7f4789beb640 -1 reset not still connected to 0x5616580681a0 2024-09-12T06:43:07.893 INFO:tasks.ceph.cluster2.osd.2.smithi152.stderr:2024-09-12T06:43:07.893+0000 7f4789beb640 -1 reset not still connected to 0x56165a282ea0 2024-09-12T06:43:08.128 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:100 IOs, 67234816 bytes 2024-09-12T06:43:10.532 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ error_code=124 2024-09-12T06:43:10.532 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set -e 2024-09-12T06:43:10.532 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T06:43:10.532 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:43:10.532 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:43:10.533 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_34 2024-09-12T06:43:10.533 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image_and_enable_mirror cluster2 mirror image_34 journal 128M 2024-09-12T06:43:10.533 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:43:10.533 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:43:10.533 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:43:10.533 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:43:10.533 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_34 2024-09-12T06:43:10.533 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:43:10.533 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local mode=journal 2024-09-12T06:43:10.533 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n journal ']' 2024-09-12T06:43:10.533 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:43:10.533 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image cluster2 mirror image_34 128M 2024-09-12T06:43:10.533 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:43:10.533 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:43:10.534 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:43:10.534 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:43:10.534 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_34 2024-09-12T06:43:10.534 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:43:10.534 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local size=128 2024-09-12T06:43:10.534 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n 128M ']' 2024-09-12T06:43:10.534 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ size=128M 2024-09-12T06:43:10.534 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:43:10.535 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_34 2024-09-12T06:43:11.193 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T06:43:11.193 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:43:11.193 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' pool = image ']' 2024-09-12T06:43:11.193 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ case "${pool}" in 2024-09-12T06:43:11.193 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 1 2024-09-12T06:43:11.193 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n '' ']' 2024-09-12T06:43:11.193 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ stress_write_image cluster2 mirror image_34 2024-09-12T06:43:11.193 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:43:11.193 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:43:11.193 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_34 2024-09-12T06:43:11.193 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T06:43:11.194 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local duration=6 2024-09-12T06:43:11.194 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set +e 2024-09-12T06:43:11.194 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_34 --debug-rbd=20 --debug-journaler=20 2024-09-12T06:43:11.328 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:11.328+0000 7fa325ae1640 -1 reset not still connected to 0x55f88b2cd5f0 2024-09-12T06:43:11.328 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:11.328+0000 7fa325ae1640 -1 reset not still connected to 0x55f896243040 2024-09-12T06:43:11.509 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:100 IOs, 1099776 bytes 2024-09-12T06:43:13.031 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:13.032+0000 7fa325ae1640 -1 reset not still connected to 0x55f85ac224e0 2024-09-12T06:43:13.033 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:13.034+0000 7fa325ae1640 -1 reset not still connected to 0x55f85ac229c0 2024-09-12T06:43:13.036 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:13.037+0000 7fa325ae1640 -1 reset not still connected to 0x55f85ba925b0 2024-09-12T06:43:13.036 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:13.037+0000 7fa325ae1640 -1 reset not still connected to 0x55f85bac29c0 2024-09-12T06:43:13.037 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:13.037+0000 7fa325ae1640 -1 reset not still connected to 0x55f85bac2b60 2024-09-12T06:43:13.037 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:13.037+0000 7fa325ae1640 -1 reset not still connected to 0x55f85cc67930 2024-09-12T06:43:13.037 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:13.037+0000 7fa325ae1640 -1 reset not still connected to 0x55f85d61d380 2024-09-12T06:43:13.037 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:13.037+0000 7fa325ae1640 -1 reset not still connected to 0x55f8636984e0 2024-09-12T06:43:13.037 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:13.037+0000 7fa325ae1640 -1 reset not still connected to 0x55f863699040 2024-09-12T06:43:13.037 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:13.037+0000 7fa325ae1640 -1 reset not still connected to 0x55f86ee2e340 2024-09-12T06:43:13.037 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:13.037+0000 7fa325ae1640 -1 reset not still connected to 0x55f87faff040 2024-09-12T06:43:13.622 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:13.623+0000 7fa325ae1640 -1 reset not still connected to 0x55f85bac3520 2024-09-12T06:43:13.626 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:13.626+0000 7fa325ae1640 -1 reset not still connected to 0x55f85cc67040 2024-09-12T06:43:13.626 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:13.626+0000 7fa325ae1640 -1 reset not still connected to 0x55f863611110 2024-09-12T06:43:13.626 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:13.626+0000 7fa325ae1640 -1 reset not still connected to 0x55f86ee2fe10 2024-09-12T06:43:13.626 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:13.626+0000 7fa325ae1640 -1 reset not still connected to 0x55f88041e270 2024-09-12T06:43:13.627 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:13.629+0000 7fa325ae1640 -1 reset not still connected to 0x55f880b4e4e0 2024-09-12T06:43:13.627 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:13.629+0000 7fa325ae1640 -1 reset not still connected to 0x55f884309c70 2024-09-12T06:43:13.627 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:13.629+0000 7fa325ae1640 -1 reset not still connected to 0x55f8849a1040 2024-09-12T06:43:13.627 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:13.629+0000 7fa325ae1640 -1 reset not still connected to 0x55f8849a11e0 2024-09-12T06:43:13.627 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:13.629+0000 7fa325ae1640 -1 reset not still connected to 0x55f8863b5790 2024-09-12T06:43:13.627 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:13.629+0000 7fa325ae1640 -1 reset not still connected to 0x55f88b2cd5f0 2024-09-12T06:43:13.627 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:13.629+0000 7fa325ae1640 -1 reset not still connected to 0x55f896243040 2024-09-12T06:43:14.171 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:200 IOs, 46467072 bytes 2024-09-12T06:43:16.313 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:43:16.314+0000 7fac12a52640 -1 reset not still connected to 0x55749e3c4d00 2024-09-12T06:43:16.313 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:43:16.315+0000 7fac12a52640 -1 reset not still connected to 0x5574b7fd3c70 2024-09-12T06:43:17.204 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ error_code=124 2024-09-12T06:43:17.205 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set -e 2024-09-12T06:43:17.205 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T06:43:17.205 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:43:17.205 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:43:17.205 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_35 2024-09-12T06:43:17.205 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image_and_enable_mirror cluster2 mirror image_35 journal 128M 2024-09-12T06:43:17.205 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:43:17.205 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:43:17.205 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:43:17.205 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:43:17.206 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_35 2024-09-12T06:43:17.206 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:43:17.206 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local mode=journal 2024-09-12T06:43:17.206 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n journal ']' 2024-09-12T06:43:17.206 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:43:17.206 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image cluster2 mirror image_35 128M 2024-09-12T06:43:17.206 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:43:17.206 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:43:17.206 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:43:17.206 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:43:17.206 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_35 2024-09-12T06:43:17.206 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:43:17.206 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local size=128 2024-09-12T06:43:17.206 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n 128M ']' 2024-09-12T06:43:17.206 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ size=128M 2024-09-12T06:43:17.207 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:43:17.207 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_35 2024-09-12T06:43:17.339 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T06:43:17.339 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:43:17.339 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' pool = image ']' 2024-09-12T06:43:17.339 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ case "${pool}" in 2024-09-12T06:43:17.339 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 1 2024-09-12T06:43:17.339 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n '' ']' 2024-09-12T06:43:17.339 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ stress_write_image cluster2 mirror image_35 2024-09-12T06:43:17.339 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:43:17.339 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:43:17.339 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_35 2024-09-12T06:43:17.339 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T06:43:17.353 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local duration=8 2024-09-12T06:43:17.353 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set +e 2024-09-12T06:43:17.353 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_35 --debug-rbd=20 --debug-journaler=20 2024-09-12T06:43:17.646 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:17.647+0000 7fa325ae1640 -1 reset not still connected to 0x55f85a4fcdd0 2024-09-12T06:43:17.646 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:17.647+0000 7fa325ae1640 -1 reset not still connected to 0x55f85bac3520 2024-09-12T06:43:17.647 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:17.648+0000 7fa325ae1640 -1 reset not still connected to 0x55f85cc67040 2024-09-12T06:43:17.647 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:17.648+0000 7fa325ae1640 -1 reset not still connected to 0x55f863611110 2024-09-12T06:43:17.647 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:17.648+0000 7fa325ae1640 -1 reset not still connected to 0x55f86ee2fe10 2024-09-12T06:43:17.651 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:17.653+0000 7fa325ae1640 -1 reset not still connected to 0x55f880b4e4e0 2024-09-12T06:43:17.654 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:17.656+0000 7fa325ae1640 -1 reset not still connected to 0x55f884309c70 2024-09-12T06:43:17.656 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:17.656+0000 7fa325ae1640 -1 reset not still connected to 0x55f8849a1040 2024-09-12T06:43:17.656 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:17.656+0000 7fa325ae1640 -1 reset not still connected to 0x55f8849a11e0 2024-09-12T06:43:17.656 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:17.656+0000 7fa325ae1640 -1 reset not still connected to 0x55f8863b5790 2024-09-12T06:43:17.657 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:17.657+0000 7fa325ae1640 -1 reset not still connected to 0x55f88b2cd5f0 2024-09-12T06:43:17.657 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:17.657+0000 7fa325ae1640 -1 reset not still connected to 0x55f896243040 2024-09-12T06:43:19.142 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:19.141+0000 7fa325ae1640 -1 reset not still connected to 0x55f85a4fcdd0 2024-09-12T06:43:19.142 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:19.141+0000 7fa325ae1640 -1 reset not still connected to 0x55f85bac3520 2024-09-12T06:43:19.143 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:19.143+0000 7fa325ae1640 -1 reset not still connected to 0x55f863611110 2024-09-12T06:43:19.144 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:19.146+0000 7fa325ae1640 -1 reset not still connected to 0x55f86ee2fe10 2024-09-12T06:43:19.144 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:19.146+0000 7fa325ae1640 -1 reset not still connected to 0x55f880b4e4e0 2024-09-12T06:43:19.146 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:19.148+0000 7fa325ae1640 -1 reset not still connected to 0x55f8849a1040 2024-09-12T06:43:19.147 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:19.148+0000 7fa325ae1640 -1 reset not still connected to 0x55f8849a11e0 2024-09-12T06:43:19.147 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:19.148+0000 7fa325ae1640 -1 reset not still connected to 0x55f8863b5790 2024-09-12T06:43:19.147 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:19.148+0000 7fa325ae1640 -1 reset not still connected to 0x55f88b2cd5f0 2024-09-12T06:43:19.147 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:19.148+0000 7fa325ae1640 -1 reset not still connected to 0x55f896243040 2024-09-12T06:43:21.488 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:43:21.488+0000 7fac12a52640 -1 reset not still connected to 0x55749b1ca5b0 2024-09-12T06:43:21.488 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:43:21.488+0000 7fac12a52640 -1 reset not still connected to 0x55749b1cbe10 2024-09-12T06:43:21.488 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:43:21.488+0000 7fac12a52640 -1 reset not still connected to 0x55749b1cbee0 2024-09-12T06:43:21.488 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:43:21.488+0000 7fac12a52640 -1 reset not still connected to 0x55749b2be0d0 2024-09-12T06:43:21.488 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:43:21.488+0000 7fac12a52640 -1 reset not still connected to 0x55749b2be8f0 2024-09-12T06:43:21.488 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:43:21.488+0000 7fac12a52640 -1 reset not still connected to 0x55749b2bed00 2024-09-12T06:43:21.488 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:43:21.488+0000 7fac12a52640 -1 reset not still connected to 0x55749b2bedd0 2024-09-12T06:43:21.488 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:43:21.488+0000 7fac12a52640 -1 reset not still connected to 0x55749b2bf6c0 2024-09-12T06:43:21.488 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:43:21.488+0000 7fac12a52640 -1 reset not still connected to 0x55749b2bfc70 2024-09-12T06:43:21.488 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:43:21.488+0000 7fac12a52640 -1 reset not still connected to 0x55749c9a7860 2024-09-12T06:43:21.488 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:43:21.488+0000 7fac12a52640 -1 reset not still connected to 0x55749ca4d790 2024-09-12T06:43:21.488 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:43:21.488+0000 7fac12a52640 -1 reset not still connected to 0x55749d4025b0 2024-09-12T06:43:21.489 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:43:21.488+0000 7fac12a52640 -1 reset not still connected to 0x55749e3c4750 2024-09-12T06:43:21.489 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:43:21.488+0000 7fac12a52640 -1 reset not still connected to 0x55749e3c48f0 2024-09-12T06:43:21.489 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:43:21.488+0000 7fac12a52640 -1 reset not still connected to 0x55749e3c4d00 2024-09-12T06:43:21.734 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:21.736+0000 7fa325ae1640 -1 reset not still connected to 0x55f863699040 2024-09-12T06:43:21.738 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:21.739+0000 7fa325ae1640 -1 reset not still connected to 0x55f86ee2e340 2024-09-12T06:43:21.741 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:21.742+0000 7fa325ae1640 -1 reset not still connected to 0x55f87faff040 2024-09-12T06:43:21.743 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:21.745+0000 7fa325ae1640 -1 reset not still connected to 0x55f88c6bfa00 2024-09-12T06:43:23.485 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:100 IOs, 81315840 bytes 2024-09-12T06:43:24.846 INFO:tasks.ceph.cluster2.osd.0.smithi152.stderr:2024-09-12T06:43:24.845+0000 7fd61689f640 -1 reset not still connected to 0x55e7657f0750 2024-09-12T06:43:24.846 INFO:tasks.ceph.cluster2.osd.0.smithi152.stderr:2024-09-12T06:43:24.845+0000 7fd61689f640 -1 reset not still connected to 0x55e76d1eba00 2024-09-12T06:43:24.846 INFO:tasks.ceph.cluster2.osd.0.smithi152.stderr:2024-09-12T06:43:24.845+0000 7fd61689f640 -1 reset not still connected to 0x55e782e41ee0 2024-09-12T06:43:24.846 INFO:tasks.ceph.cluster2.osd.0.smithi152.stderr:2024-09-12T06:43:24.845+0000 7fd61689f640 -1 reset not still connected to 0x55e785c50b60 2024-09-12T06:43:24.847 INFO:tasks.ceph.cluster2.osd.0.smithi152.stderr:2024-09-12T06:43:24.845+0000 7fd61689f640 -1 reset not still connected to 0x55e785c51520 2024-09-12T06:43:25.362 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ error_code=124 2024-09-12T06:43:25.363 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set -e 2024-09-12T06:43:25.363 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T06:43:25.363 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:43:25.363 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:43:25.363 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_36 2024-09-12T06:43:25.363 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image_and_enable_mirror cluster2 mirror image_36 journal 128M 2024-09-12T06:43:25.363 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:43:25.363 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:43:25.363 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:43:25.363 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:43:25.363 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_36 2024-09-12T06:43:25.363 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:43:25.363 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local mode=journal 2024-09-12T06:43:25.364 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n journal ']' 2024-09-12T06:43:25.364 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:43:25.364 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image cluster2 mirror image_36 128M 2024-09-12T06:43:25.364 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:43:25.364 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:43:25.364 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:43:25.364 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:43:25.364 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_36 2024-09-12T06:43:25.364 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:43:25.364 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local size=128 2024-09-12T06:43:25.364 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n 128M ']' 2024-09-12T06:43:25.364 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ size=128M 2024-09-12T06:43:25.365 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:43:25.365 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_36 2024-09-12T06:43:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T06:43:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:43:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' pool = image ']' 2024-09-12T06:43:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ case "${pool}" in 2024-09-12T06:43:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 1 2024-09-12T06:43:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n '' ']' 2024-09-12T06:43:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ stress_write_image cluster2 mirror image_36 2024-09-12T06:43:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:43:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:43:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_36 2024-09-12T06:43:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T06:43:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local duration=7 2024-09-12T06:43:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set +e 2024-09-12T06:43:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_36 --debug-rbd=20 --debug-journaler=20 2024-09-12T06:43:26.842 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:100 IOs, 42931200 bytes 2024-09-12T06:43:28.951 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:28.951+0000 7fa325ae1640 -1 reset not still connected to 0x55f85cc67860 2024-09-12T06:43:28.952 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:28.953+0000 7fa325ae1640 -1 reset not still connected to 0x55f85cc67930 2024-09-12T06:43:28.952 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:28.953+0000 7fa325ae1640 -1 reset not still connected to 0x55f85d61d380 2024-09-12T06:43:28.952 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:28.953+0000 7fa325ae1640 -1 reset not still connected to 0x55f8636984e0 2024-09-12T06:43:28.952 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:28.953+0000 7fa325ae1640 -1 reset not still connected to 0x55f863699040 2024-09-12T06:43:28.952 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:28.953+0000 7fa325ae1640 -1 reset not still connected to 0x55f86ee2e340 2024-09-12T06:43:28.952 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:28.953+0000 7fa325ae1640 -1 reset not still connected to 0x55f87faff040 2024-09-12T06:43:28.952 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:28.953+0000 7fa325ae1640 -1 reset not still connected to 0x55f88c6bfa00 2024-09-12T06:43:30.094 INFO:tasks.ceph.cluster2.osd.2.smithi152.stderr:2024-09-12T06:43:30.096+0000 7f4789beb640 -1 reset not still connected to 0x56163e3b3d40 2024-09-12T06:43:30.095 INFO:tasks.ceph.cluster2.osd.2.smithi152.stderr:2024-09-12T06:43:30.096+0000 7f4789beb640 -1 reset not still connected to 0x56163e3b3ee0 2024-09-12T06:43:30.258 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:30.260+0000 7fa325ae1640 -1 reset not still connected to 0x55f85a4fcdd0 2024-09-12T06:43:30.262 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:30.263+0000 7fa325ae1640 -1 reset not still connected to 0x55f85bac3520 2024-09-12T06:43:30.262 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:30.263+0000 7fa325ae1640 -1 reset not still connected to 0x55f85cc67040 2024-09-12T06:43:30.262 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:30.263+0000 7fa325ae1640 -1 reset not still connected to 0x55f863611110 2024-09-12T06:43:30.262 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:30.263+0000 7fa325ae1640 -1 reset not still connected to 0x55f86ee2fe10 2024-09-12T06:43:30.262 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:30.263+0000 7fa325ae1640 -1 reset not still connected to 0x55f880b4e4e0 2024-09-12T06:43:30.263 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:30.263+0000 7fa325ae1640 -1 reset not still connected to 0x55f884309c70 2024-09-12T06:43:30.266 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:30.267+0000 7fa325ae1640 -1 reset not still connected to 0x55f8849a1040 2024-09-12T06:43:30.266 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:30.267+0000 7fa325ae1640 -1 reset not still connected to 0x55f8849a11e0 2024-09-12T06:43:30.266 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:30.267+0000 7fa325ae1640 -1 reset not still connected to 0x55f8863b4680 2024-09-12T06:43:30.266 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:30.267+0000 7fa325ae1640 -1 reset not still connected to 0x55f8863b5790 2024-09-12T06:43:30.266 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:30.267+0000 7fa325ae1640 -1 reset not still connected to 0x55f88b2cd5f0 2024-09-12T06:43:30.267 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:30.267+0000 7fa325ae1640 -1 reset not still connected to 0x55f896243040 2024-09-12T06:43:30.305 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:30.305+0000 7fa325ae1640 -1 reset not still connected to 0x55f85a4fcdd0 2024-09-12T06:43:30.305 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:30.305+0000 7fa325ae1640 -1 reset not still connected to 0x55f85bac3520 2024-09-12T06:43:30.306 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:30.307+0000 7fa325ae1640 -1 reset not still connected to 0x55f85cc67040 2024-09-12T06:43:30.306 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:30.307+0000 7fa325ae1640 -1 reset not still connected to 0x55f863611110 2024-09-12T06:43:30.307 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:30.307+0000 7fa325ae1640 -1 reset not still connected to 0x55f86ee2fe10 2024-09-12T06:43:30.307 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:30.307+0000 7fa325ae1640 -1 reset not still connected to 0x55f880b4e4e0 2024-09-12T06:43:30.308 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:30.310+0000 7fa325ae1640 -1 reset not still connected to 0x55f8849a1040 2024-09-12T06:43:30.308 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:30.310+0000 7fa325ae1640 -1 reset not still connected to 0x55f8849a11e0 2024-09-12T06:43:30.308 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:30.310+0000 7fa325ae1640 -1 reset not still connected to 0x55f8863b4680 2024-09-12T06:43:30.308 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:30.310+0000 7fa325ae1640 -1 reset not still connected to 0x55f8863b5790 2024-09-12T06:43:30.308 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:30.310+0000 7fa325ae1640 -1 reset not still connected to 0x55f88b2cd5f0 2024-09-12T06:43:30.309 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:30.310+0000 7fa325ae1640 -1 reset not still connected to 0x55f896243040 2024-09-12T06:43:30.686 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:30.687+0000 7fa325ae1640 -1 reset not still connected to 0x55f85bac29c0 2024-09-12T06:43:30.689 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:30.691+0000 7fa325ae1640 -1 reset not still connected to 0x55f85bac2b60 2024-09-12T06:43:30.689 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:30.691+0000 7fa325ae1640 -1 reset not still connected to 0x55f85cc67860 2024-09-12T06:43:30.689 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:30.691+0000 7fa325ae1640 -1 reset not still connected to 0x55f85cc67930 2024-09-12T06:43:30.689 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:30.691+0000 7fa325ae1640 -1 reset not still connected to 0x55f85d61d380 2024-09-12T06:43:30.689 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:30.691+0000 7fa325ae1640 -1 reset not still connected to 0x55f8636984e0 2024-09-12T06:43:30.689 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:30.691+0000 7fa325ae1640 -1 reset not still connected to 0x55f863699040 2024-09-12T06:43:30.689 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:30.691+0000 7fa325ae1640 -1 reset not still connected to 0x55f86ee2e340 2024-09-12T06:43:30.690 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:30.691+0000 7fa325ae1640 -1 reset not still connected to 0x55f87faff040 2024-09-12T06:43:30.690 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:30.691+0000 7fa325ae1640 -1 reset not still connected to 0x55f88c6bfa00 2024-09-12T06:43:30.861 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:43:30.862+0000 7fac12a52640 -1 reset not still connected to 0x55749b2bfc70 2024-09-12T06:43:30.861 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:43:30.862+0000 7fac12a52640 -1 reset not still connected to 0x55749c9a7860 2024-09-12T06:43:30.861 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:43:30.862+0000 7fac12a52640 -1 reset not still connected to 0x55749ca4d790 2024-09-12T06:43:30.864 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:43:30.865+0000 7fac12a52640 -1 reset not still connected to 0x55749d4025b0 2024-09-12T06:43:30.864 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:43:30.865+0000 7fac12a52640 -1 reset not still connected to 0x55749e3c4750 2024-09-12T06:43:30.864 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:43:30.865+0000 7fac12a52640 -1 reset not still connected to 0x55749e3c48f0 2024-09-12T06:43:30.864 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:43:30.865+0000 7fac12a52640 -1 reset not still connected to 0x55749e3c4d00 2024-09-12T06:43:31.745 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:43:31.746+0000 7fac12a52640 -1 reset not still connected to 0x55749e3c4750 2024-09-12T06:43:31.745 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:43:31.746+0000 7fac12a52640 -1 reset not still connected to 0x55749e3c4d00 2024-09-12T06:43:31.749 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:43:31.751+0000 7fac12a52640 -1 reset not still connected to 0x5574b7fd3c70 2024-09-12T06:43:31.749 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:43:31.751+0000 7fac12a52640 -1 reset not still connected to 0x5574cfee0680 2024-09-12T06:43:32.178 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:200 IOs, 96480256 bytes 2024-09-12T06:43:32.484 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ error_code=124 2024-09-12T06:43:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set -e 2024-09-12T06:43:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T06:43:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:43:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:43:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_37 2024-09-12T06:43:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image_and_enable_mirror cluster2 mirror image_37 journal 128M 2024-09-12T06:43:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:43:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:43:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:43:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:43:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_37 2024-09-12T06:43:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:43:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local mode=journal 2024-09-12T06:43:32.486 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n journal ']' 2024-09-12T06:43:32.486 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:43:32.486 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image cluster2 mirror image_37 128M 2024-09-12T06:43:32.486 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:43:32.486 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:43:32.486 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:43:32.486 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:43:32.486 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_37 2024-09-12T06:43:32.486 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:43:32.486 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local size=128 2024-09-12T06:43:32.486 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n 128M ']' 2024-09-12T06:43:32.486 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ size=128M 2024-09-12T06:43:32.486 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:43:32.486 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_37 2024-09-12T06:43:32.560 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T06:43:32.561 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:43:32.561 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' pool = image ']' 2024-09-12T06:43:32.561 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ case "${pool}" in 2024-09-12T06:43:32.561 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 1 2024-09-12T06:43:32.561 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n '' ']' 2024-09-12T06:43:32.561 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ stress_write_image cluster2 mirror image_37 2024-09-12T06:43:32.561 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:43:32.561 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:43:32.561 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_37 2024-09-12T06:43:32.561 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T06:43:32.562 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local duration=7 2024-09-12T06:43:32.562 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set +e 2024-09-12T06:43:32.562 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_37 --debug-rbd=20 --debug-journaler=20 2024-09-12T06:43:33.411 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:100 IOs, 7918592 bytes 2024-09-12T06:43:34.876 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:43:34.878+0000 7fac12a52640 -1 reset not still connected to 0x5574d5072c30 2024-09-12T06:43:36.269 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:200 IOs, 62014464 bytes 2024-09-12T06:43:36.854 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:43:36.856+0000 7fac12a52640 -1 reset not still connected to 0x55749d402410 2024-09-12T06:43:36.855 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:43:36.856+0000 7fac12a52640 -1 reset not still connected to 0x55749d403ba0 2024-09-12T06:43:36.855 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:43:36.856+0000 7fac12a52640 -1 reset not still connected to 0x55749d7ca270 2024-09-12T06:43:36.857 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:43:36.858+0000 7fac12a52640 -1 reset not still connected to 0x5574a2fa7a00 2024-09-12T06:43:36.862 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:43:36.864+0000 7fac12a52640 -1 reset not still connected to 0x5574bdabf1e0 2024-09-12T06:43:36.862 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:43:36.864+0000 7fac12a52640 -1 reset not still connected to 0x5574d5072c30 2024-09-12T06:43:39.408 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:43:39.408+0000 7fac12a52640 -1 reset not still connected to 0x5574d5072c30 2024-09-12T06:43:39.576 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ error_code=124 2024-09-12T06:43:39.576 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set -e 2024-09-12T06:43:39.576 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T06:43:39.576 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:43:39.576 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:43:39.576 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_38 2024-09-12T06:43:39.576 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image_and_enable_mirror cluster2 mirror image_38 journal 128M 2024-09-12T06:43:39.576 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:43:39.576 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:43:39.577 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:43:39.577 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:43:39.577 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_38 2024-09-12T06:43:39.577 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:43:39.577 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local mode=journal 2024-09-12T06:43:39.577 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n journal ']' 2024-09-12T06:43:39.577 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:43:39.577 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image cluster2 mirror image_38 128M 2024-09-12T06:43:39.577 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:43:39.577 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:43:39.577 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:43:39.577 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:43:39.577 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_38 2024-09-12T06:43:39.577 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:43:39.578 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local size=128 2024-09-12T06:43:39.578 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n 128M ']' 2024-09-12T06:43:39.578 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ size=128M 2024-09-12T06:43:39.578 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:43:39.578 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_38 2024-09-12T06:43:39.653 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T06:43:39.653 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:43:39.653 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' pool = image ']' 2024-09-12T06:43:39.653 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ case "${pool}" in 2024-09-12T06:43:39.653 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 1 2024-09-12T06:43:39.653 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n '' ']' 2024-09-12T06:43:39.653 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ stress_write_image cluster2 mirror image_38 2024-09-12T06:43:39.653 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:43:39.653 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:43:39.654 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_38 2024-09-12T06:43:39.654 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T06:43:39.654 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local duration=5 2024-09-12T06:43:39.654 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set +e 2024-09-12T06:43:39.654 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_38 --debug-rbd=20 --debug-journaler=20 2024-09-12T06:43:40.577 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:43:40.579+0000 7f80d15bb640 -1 reset not still connected to 0x556b33ff76c0 2024-09-12T06:43:40.758 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:43:40.758+0000 7f80d15bb640 -1 reset not still connected to 0x556b33ff76c0 2024-09-12T06:43:41.802 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:100 IOs, 34515968 bytes 2024-09-12T06:43:42.133 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:43:42.135+0000 7fac12a52640 -1 reset not still connected to 0x55749b2be0d0 2024-09-12T06:43:42.133 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:43:42.135+0000 7fac12a52640 -1 reset not still connected to 0x55749b2be8f0 2024-09-12T06:43:42.136 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:43:42.137+0000 7fac12a52640 -1 reset not still connected to 0x55749b2bed00 2024-09-12T06:43:42.136 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:43:42.137+0000 7fac12a52640 -1 reset not still connected to 0x55749b2bedd0 2024-09-12T06:43:42.136 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:43:42.137+0000 7fac12a52640 -1 reset not still connected to 0x55749b2bf6c0 2024-09-12T06:43:42.136 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:43:42.137+0000 7fac12a52640 -1 reset not still connected to 0x55749b2bfc70 2024-09-12T06:43:42.136 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:43:42.137+0000 7fac12a52640 -1 reset not still connected to 0x55749c9a7860 2024-09-12T06:43:42.142 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:43:42.143+0000 7fac12a52640 -1 reset not still connected to 0x55749d3c4750 2024-09-12T06:43:42.143 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:43:42.143+0000 7fac12a52640 -1 reset not still connected to 0x55749d4025b0 2024-09-12T06:43:42.143 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:43:42.143+0000 7fac12a52640 -1 reset not still connected to 0x55749e3c4750 2024-09-12T06:43:42.143 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:43:42.143+0000 7fac12a52640 -1 reset not still connected to 0x55749e3c48f0 2024-09-12T06:43:42.143 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:43:42.143+0000 7fac12a52640 -1 reset not still connected to 0x55749e3c4d00 2024-09-12T06:43:44.025 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:43:44.025+0000 7fac12a52640 -1 reset not still connected to 0x55749b1cbe10 2024-09-12T06:43:44.025 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:43:44.025+0000 7fac12a52640 -1 reset not still connected to 0x55749b1cbee0 2024-09-12T06:43:44.025 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:43:44.025+0000 7fac12a52640 -1 reset not still connected to 0x55749b2be0d0 2024-09-12T06:43:44.025 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:43:44.025+0000 7fac12a52640 -1 reset not still connected to 0x55749b2be8f0 2024-09-12T06:43:44.025 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:43:44.025+0000 7fac12a52640 -1 reset not still connected to 0x55749b2bed00 2024-09-12T06:43:44.025 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:43:44.025+0000 7fac12a52640 -1 reset not still connected to 0x55749b2bedd0 2024-09-12T06:43:44.025 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:43:44.025+0000 7fac12a52640 -1 reset not still connected to 0x55749b2bf6c0 2024-09-12T06:43:44.025 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:43:44.025+0000 7fac12a52640 -1 reset not still connected to 0x55749b2bfc70 2024-09-12T06:43:44.025 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:43:44.025+0000 7fac12a52640 -1 reset not still connected to 0x55749c9a7860 2024-09-12T06:43:44.026 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:43:44.025+0000 7fac12a52640 -1 reset not still connected to 0x55749ca4d790 2024-09-12T06:43:44.026 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:43:44.025+0000 7fac12a52640 -1 reset not still connected to 0x55749d3c4750 2024-09-12T06:43:44.026 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:43:44.025+0000 7fac12a52640 -1 reset not still connected to 0x55749d4025b0 2024-09-12T06:43:44.026 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:43:44.026+0000 7fac12a52640 -1 reset not still connected to 0x55749e3c4750 2024-09-12T06:43:44.026 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:43:44.026+0000 7fac12a52640 -1 reset not still connected to 0x55749e3c48f0 2024-09-12T06:43:44.026 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:43:44.026+0000 7fac12a52640 -1 reset not still connected to 0x55749e3c4d00 2024-09-12T06:43:44.029 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:43:44.031+0000 7fac12a52640 -1 reset not still connected to 0x5574b7fd3c70 2024-09-12T06:43:44.029 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:43:44.031+0000 7fac12a52640 -1 reset not still connected to 0x5574cfee0680 2024-09-12T06:43:44.066 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:200 IOs, 73059328 bytes 2024-09-12T06:43:44.665 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ error_code=124 2024-09-12T06:43:44.665 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set -e 2024-09-12T06:43:44.665 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T06:43:44.665 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:43:44.665 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:43:44.666 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_39 2024-09-12T06:43:44.666 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image_and_enable_mirror cluster2 mirror image_39 journal 128M 2024-09-12T06:43:44.666 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:43:44.666 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:43:44.666 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:43:44.666 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:43:44.666 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_39 2024-09-12T06:43:44.666 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:43:44.666 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local mode=journal 2024-09-12T06:43:44.666 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n journal ']' 2024-09-12T06:43:44.666 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:43:44.666 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image cluster2 mirror image_39 128M 2024-09-12T06:43:44.666 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:43:44.666 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:43:44.667 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:43:44.667 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:43:44.667 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_39 2024-09-12T06:43:44.667 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:43:44.667 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local size=128 2024-09-12T06:43:44.667 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n 128M ']' 2024-09-12T06:43:44.667 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ size=128M 2024-09-12T06:43:44.667 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:43:44.667 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_39 2024-09-12T06:43:44.749 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T06:43:44.749 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:43:44.749 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' pool = image ']' 2024-09-12T06:43:44.749 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ case "${pool}" in 2024-09-12T06:43:44.749 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 1 2024-09-12T06:43:44.749 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n '' ']' 2024-09-12T06:43:44.749 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ stress_write_image cluster2 mirror image_39 2024-09-12T06:43:44.749 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:43:44.749 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:43:44.749 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_39 2024-09-12T06:43:44.749 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T06:43:44.750 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local duration=14 2024-09-12T06:43:44.750 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set +e 2024-09-12T06:43:44.750 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_39 --debug-rbd=20 --debug-journaler=20 2024-09-12T06:43:46.956 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:100 IOs, 27826176 bytes 2024-09-12T06:43:47.154 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:43:47.154+0000 7fac12a52640 -1 reset not still connected to 0x5574d5072c30 2024-09-12T06:43:51.844 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:200 IOs, 100546560 bytes 2024-09-12T06:43:52.724 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:52.724+0000 7fa325ae1640 -1 reset not still connected to 0x55f85a4fcd00 2024-09-12T06:43:52.725 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:52.724+0000 7fa325ae1640 -1 reset not still connected to 0x55f85a4fcdd0 2024-09-12T06:43:52.725 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:52.724+0000 7fa325ae1640 -1 reset not still connected to 0x55f85bac3520 2024-09-12T06:43:52.725 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:52.724+0000 7fa325ae1640 -1 reset not still connected to 0x55f85cc67040 2024-09-12T06:43:52.725 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:52.724+0000 7fa325ae1640 -1 reset not still connected to 0x55f863611110 2024-09-12T06:43:52.725 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:52.724+0000 7fa325ae1640 -1 reset not still connected to 0x55f86ee2fe10 2024-09-12T06:43:52.725 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:52.725+0000 7fa325ae1640 -1 reset not still connected to 0x55f88041e270 2024-09-12T06:43:52.725 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:52.725+0000 7fa325ae1640 -1 reset not still connected to 0x55f880b4e4e0 2024-09-12T06:43:52.725 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:52.725+0000 7fa325ae1640 -1 reset not still connected to 0x55f8849a1040 2024-09-12T06:43:52.725 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:52.725+0000 7fa325ae1640 -1 reset not still connected to 0x55f8849a11e0 2024-09-12T06:43:52.725 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:52.725+0000 7fa325ae1640 -1 reset not still connected to 0x55f8863b4680 2024-09-12T06:43:52.726 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:52.725+0000 7fa325ae1640 -1 reset not still connected to 0x55f8863b5790 2024-09-12T06:43:52.726 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:52.726+0000 7fa325ae1640 -1 reset not still connected to 0x55f88b2cd5f0 2024-09-12T06:43:52.726 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:52.726+0000 7fa325ae1640 -1 reset not still connected to 0x55f8926ce0d0 2024-09-12T06:43:52.726 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:52.726+0000 7fa325ae1640 -1 reset not still connected to 0x55f896243040 2024-09-12T06:43:52.829 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:300 IOs, 111021056 bytes 2024-09-12T06:43:53.314 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:43:53.316+0000 7f80d15bb640 -1 reset not still connected to 0x556b02ee8dd0 2024-09-12T06:43:53.315 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:43:53.316+0000 7f80d15bb640 -1 reset not still connected to 0x556b030a2c30 2024-09-12T06:43:54.447 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:54.448+0000 7fa325ae1640 -1 reset not still connected to 0x55f8636984e0 2024-09-12T06:43:54.447 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:54.448+0000 7fa325ae1640 -1 reset not still connected to 0x55f863699040 2024-09-12T06:43:54.447 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:54.448+0000 7fa325ae1640 -1 reset not still connected to 0x55f86ee2e340 2024-09-12T06:43:54.447 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:54.448+0000 7fa325ae1640 -1 reset not still connected to 0x55f87faff040 2024-09-12T06:43:54.447 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:54.448+0000 7fa325ae1640 -1 reset not still connected to 0x55f88c6bfa00 2024-09-12T06:43:54.447 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:43:54.448+0000 7fa325ae1640 -1 reset not still connected to 0x55f8927168f0 2024-09-12T06:43:56.512 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:400 IOs, 167565312 bytes 2024-09-12T06:43:58.762 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ error_code=124 2024-09-12T06:43:58.762 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set -e 2024-09-12T06:43:58.762 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T06:43:58.762 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:43:58.762 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:43:58.762 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_40 2024-09-12T06:43:58.762 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image_and_enable_mirror cluster2 mirror image_40 journal 128M 2024-09-12T06:43:58.762 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:43:58.762 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:43:58.763 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:43:58.763 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:43:58.763 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_40 2024-09-12T06:43:58.763 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:43:58.763 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local mode=journal 2024-09-12T06:43:58.763 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n journal ']' 2024-09-12T06:43:58.763 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:43:58.763 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image cluster2 mirror image_40 128M 2024-09-12T06:43:58.763 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:43:58.763 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:43:58.763 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:43:58.763 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:43:58.763 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_40 2024-09-12T06:43:58.763 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:43:58.764 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local size=128 2024-09-12T06:43:58.764 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n 128M ']' 2024-09-12T06:43:58.764 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ size=128M 2024-09-12T06:43:58.764 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:43:58.764 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_40 2024-09-12T06:43:58.984 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T06:43:58.984 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:43:58.984 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' pool = image ']' 2024-09-12T06:43:58.984 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ case "${pool}" in 2024-09-12T06:43:58.984 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 1 2024-09-12T06:43:58.984 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n '' ']' 2024-09-12T06:43:58.984 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ stress_write_image cluster2 mirror image_40 2024-09-12T06:43:58.984 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:43:58.984 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:43:58.984 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_40 2024-09-12T06:43:58.984 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T06:43:58.985 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local duration=9 2024-09-12T06:43:58.986 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set +e 2024-09-12T06:43:58.986 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_40 --debug-rbd=20 --debug-journaler=20 2024-09-12T06:44:03.044 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:100 IOs, 55848960 bytes 2024-09-12T06:44:03.754 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:03.755+0000 7fac12a52640 -1 reset not still connected to 0x5574a2fa7a00 2024-09-12T06:44:03.755 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:03.756+0000 7fac12a52640 -1 reset not still connected to 0x5574bdabf1e0 2024-09-12T06:44:03.755 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:03.756+0000 7fac12a52640 -1 reset not still connected to 0x5574d5072c30 2024-09-12T06:44:05.141 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:05.143+0000 7fa325ae1640 -1 reset not still connected to 0x55f859179380 2024-09-12T06:44:05.143 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:05.144+0000 7fa325ae1640 -1 reset not still connected to 0x55f85a304a90 2024-09-12T06:44:05.143 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:05.144+0000 7fa325ae1640 -1 reset not still connected to 0x55f85ac224e0 2024-09-12T06:44:05.143 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:05.144+0000 7fa325ae1640 -1 reset not still connected to 0x55f85ac229c0 2024-09-12T06:44:05.145 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:05.146+0000 7fa325ae1640 -1 reset not still connected to 0x55f85ba925b0 2024-09-12T06:44:05.145 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:05.146+0000 7fa325ae1640 -1 reset not still connected to 0x55f85bac29c0 2024-09-12T06:44:05.145 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:05.146+0000 7fa325ae1640 -1 reset not still connected to 0x55f85bac2b60 2024-09-12T06:44:05.148 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:05.148+0000 7fa325ae1640 -1 reset not still connected to 0x55f85cc67860 2024-09-12T06:44:05.148 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:05.148+0000 7fa325ae1640 -1 reset not still connected to 0x55f85cc67930 2024-09-12T06:44:05.148 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:05.148+0000 7fa325ae1640 -1 reset not still connected to 0x55f85d61d380 2024-09-12T06:44:05.148 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:05.149+0000 7fa325ae1640 -1 reset not still connected to 0x55f8636984e0 2024-09-12T06:44:05.148 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:05.149+0000 7fa325ae1640 -1 reset not still connected to 0x55f863699040 2024-09-12T06:44:05.148 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:05.149+0000 7fa325ae1640 -1 reset not still connected to 0x55f86ee2e340 2024-09-12T06:44:05.149 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:05.149+0000 7fa325ae1640 -1 reset not still connected to 0x55f87faff040 2024-09-12T06:44:05.149 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:05.149+0000 7fa325ae1640 -1 reset not still connected to 0x55f88c6bfa00 2024-09-12T06:44:05.149 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:05.149+0000 7fa325ae1640 -1 reset not still connected to 0x55f8927168f0 2024-09-12T06:44:05.867 INFO:tasks.ceph.cluster2.osd.0.smithi152.stderr:2024-09-12T06:44:05.864+0000 7fd61689f640 -1 reset not still connected to 0x55e77b1fcc30 2024-09-12T06:44:05.867 INFO:tasks.ceph.cluster2.osd.0.smithi152.stderr:2024-09-12T06:44:05.868+0000 7fd61689f640 -1 reset not still connected to 0x55e77c768680 2024-09-12T06:44:05.871 INFO:tasks.ceph.cluster2.osd.0.smithi152.stderr:2024-09-12T06:44:05.873+0000 7fd61689f640 -1 reset not still connected to 0x55e77d6f2d00 2024-09-12T06:44:05.872 INFO:tasks.ceph.cluster2.osd.0.smithi152.stderr:2024-09-12T06:44:05.873+0000 7fd61689f640 -1 reset not still connected to 0x55e77d6f3a00 2024-09-12T06:44:05.872 INFO:tasks.ceph.cluster2.osd.0.smithi152.stderr:2024-09-12T06:44:05.873+0000 7fd61689f640 -1 reset not still connected to 0x55e782f70750 2024-09-12T06:44:06.233 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:200 IOs, 103569408 bytes 2024-09-12T06:44:06.544 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:300 IOs, 114215936 bytes 2024-09-12T06:44:07.998 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ error_code=124 2024-09-12T06:44:07.998 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set -e 2024-09-12T06:44:07.998 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T06:44:07.998 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:44:07.998 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:44:07.998 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_41 2024-09-12T06:44:07.998 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image_and_enable_mirror cluster2 mirror image_41 journal 128M 2024-09-12T06:44:07.998 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:44:07.998 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:44:07.998 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:44:07.998 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:44:07.999 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_41 2024-09-12T06:44:07.999 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:44:07.999 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local mode=journal 2024-09-12T06:44:07.999 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n journal ']' 2024-09-12T06:44:07.999 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:44:07.999 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image cluster2 mirror image_41 128M 2024-09-12T06:44:07.999 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:44:07.999 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:44:07.999 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:44:07.999 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:44:07.999 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_41 2024-09-12T06:44:07.999 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:44:07.999 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local size=128 2024-09-12T06:44:08.000 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n 128M ']' 2024-09-12T06:44:08.000 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ size=128M 2024-09-12T06:44:08.000 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:44:08.000 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_41 2024-09-12T06:44:08.080 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T06:44:08.080 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:44:08.080 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' pool = image ']' 2024-09-12T06:44:08.080 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ case "${pool}" in 2024-09-12T06:44:08.080 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 1 2024-09-12T06:44:08.080 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n '' ']' 2024-09-12T06:44:08.081 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ stress_write_image cluster2 mirror image_41 2024-09-12T06:44:08.081 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:44:08.081 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:44:08.081 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_41 2024-09-12T06:44:08.081 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T06:44:08.082 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local duration=5 2024-09-12T06:44:08.082 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set +e 2024-09-12T06:44:08.082 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_41 --debug-rbd=20 --debug-journaler=20 2024-09-12T06:44:12.869 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:100 IOs, 56459264 bytes 2024-09-12T06:44:13.094 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ error_code=124 2024-09-12T06:44:13.094 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set -e 2024-09-12T06:44:13.094 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T06:44:13.094 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:44:13.094 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:44:13.094 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_42 2024-09-12T06:44:13.094 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image_and_enable_mirror cluster2 mirror image_42 journal 128M 2024-09-12T06:44:13.094 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:44:13.094 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:44:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:44:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:44:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_42 2024-09-12T06:44:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:44:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local mode=journal 2024-09-12T06:44:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n journal ']' 2024-09-12T06:44:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:44:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image cluster2 mirror image_42 128M 2024-09-12T06:44:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:44:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:44:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:44:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:44:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_42 2024-09-12T06:44:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:44:13.096 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local size=128 2024-09-12T06:44:13.096 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n 128M ']' 2024-09-12T06:44:13.096 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ size=128M 2024-09-12T06:44:13.096 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:44:13.096 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_42 2024-09-12T06:44:13.176 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T06:44:13.177 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:44:13.177 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' pool = image ']' 2024-09-12T06:44:13.177 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ case "${pool}" in 2024-09-12T06:44:13.177 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 1 2024-09-12T06:44:13.177 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n '' ']' 2024-09-12T06:44:13.177 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ stress_write_image cluster2 mirror image_42 2024-09-12T06:44:13.177 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:44:13.178 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:44:13.178 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_42 2024-09-12T06:44:13.178 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T06:44:13.179 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local duration=9 2024-09-12T06:44:13.179 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set +e 2024-09-12T06:44:13.179 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_42 --debug-rbd=20 --debug-journaler=20 2024-09-12T06:44:15.871 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:100 IOs, 41061376 bytes 2024-09-12T06:44:16.737 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:200 IOs, 58375168 bytes 2024-09-12T06:44:18.401 INFO:tasks.ceph.cluster2.osd.0.smithi152.stderr:2024-09-12T06:44:18.402+0000 7fd61689f640 -1 reset not still connected to 0x55e7657f0750 2024-09-12T06:44:18.401 INFO:tasks.ceph.cluster2.osd.0.smithi152.stderr:2024-09-12T06:44:18.402+0000 7fd61689f640 -1 reset not still connected to 0x55e76d1eba00 2024-09-12T06:44:18.401 INFO:tasks.ceph.cluster2.osd.0.smithi152.stderr:2024-09-12T06:44:18.402+0000 7fd61689f640 -1 reset not still connected to 0x55e782e41ee0 2024-09-12T06:44:18.401 INFO:tasks.ceph.cluster2.osd.0.smithi152.stderr:2024-09-12T06:44:18.402+0000 7fd61689f640 -1 reset not still connected to 0x55e785c51520 2024-09-12T06:44:18.941 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:300 IOs, 90808320 bytes 2024-09-12T06:44:19.192 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:19.194+0000 7fa325ae1640 -1 reset not still connected to 0x55f85a3bab60 2024-09-12T06:44:19.199 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:19.201+0000 7fa325ae1640 -1 reset not still connected to 0x55f85a4fc9c0 2024-09-12T06:44:19.201 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:19.201+0000 7fa325ae1640 -1 reset not still connected to 0x55f85a4fcd00 2024-09-12T06:44:19.201 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:19.201+0000 7fa325ae1640 -1 reset not still connected to 0x55f85a4fcdd0 2024-09-12T06:44:19.201 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:19.202+0000 7fa325ae1640 -1 reset not still connected to 0x55f85bac3520 2024-09-12T06:44:19.202 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:19.203+0000 7fa325ae1640 -1 reset not still connected to 0x55f85cc67040 2024-09-12T06:44:19.202 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:19.203+0000 7fa325ae1640 -1 reset not still connected to 0x55f863611110 2024-09-12T06:44:19.202 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:19.203+0000 7fa325ae1640 -1 reset not still connected to 0x55f869fdc0d0 2024-09-12T06:44:19.202 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:19.203+0000 7fa325ae1640 -1 reset not still connected to 0x55f86ee2fe10 2024-09-12T06:44:19.202 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:19.203+0000 7fa325ae1640 -1 reset not still connected to 0x55f88041e270 2024-09-12T06:44:19.202 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:19.203+0000 7fa325ae1640 -1 reset not still connected to 0x55f880b4e4e0 2024-09-12T06:44:19.202 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:19.203+0000 7fa325ae1640 -1 reset not still connected to 0x55f884309c70 2024-09-12T06:44:19.202 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:19.203+0000 7fa325ae1640 -1 reset not still connected to 0x55f8849a1040 2024-09-12T06:44:19.202 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:19.203+0000 7fa325ae1640 -1 reset not still connected to 0x55f8849a11e0 2024-09-12T06:44:19.202 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:19.203+0000 7fa325ae1640 -1 reset not still connected to 0x55f8863b4340 2024-09-12T06:44:19.203 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:19.203+0000 7fa325ae1640 -1 reset not still connected to 0x55f8863b4680 2024-09-12T06:44:19.203 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:19.203+0000 7fa325ae1640 -1 reset not still connected to 0x55f8863b5790 2024-09-12T06:44:19.203 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:19.203+0000 7fa325ae1640 -1 reset not still connected to 0x55f88b2cd5f0 2024-09-12T06:44:19.203 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:19.203+0000 7fa325ae1640 -1 reset not still connected to 0x55f8926ce0d0 2024-09-12T06:44:19.203 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:19.203+0000 7fa325ae1640 -1 reset not still connected to 0x55f896243040 2024-09-12T06:44:19.863 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:19.862+0000 7fa325ae1640 -1 reset not still connected to 0x55f896243040 2024-09-12T06:44:19.937 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:19.937+0000 7fa325ae1640 -1 reset not still connected to 0x55f86ee2e340 2024-09-12T06:44:19.937 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:19.937+0000 7fa325ae1640 -1 reset not still connected to 0x55f87faff040 2024-09-12T06:44:19.937 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:19.937+0000 7fa325ae1640 -1 reset not still connected to 0x55f88c6bfa00 2024-09-12T06:44:19.942 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:19.943+0000 7fa325ae1640 -1 reset not still connected to 0x55f8927168f0 2024-09-12T06:44:20.846 INFO:tasks.ceph.cluster2.osd.2.smithi152.stderr:2024-09-12T06:44:20.845+0000 7f4789beb640 -1 reset not still connected to 0x561658359790 2024-09-12T06:44:22.190 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ error_code=124 2024-09-12T06:44:22.190 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set -e 2024-09-12T06:44:22.190 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T06:44:22.190 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:44:22.190 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:44:22.190 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_43 2024-09-12T06:44:22.190 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image_and_enable_mirror cluster2 mirror image_43 journal 128M 2024-09-12T06:44:22.190 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:44:22.190 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:44:22.190 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:44:22.191 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:44:22.191 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_43 2024-09-12T06:44:22.191 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:44:22.191 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local mode=journal 2024-09-12T06:44:22.191 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n journal ']' 2024-09-12T06:44:22.191 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:44:22.191 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image cluster2 mirror image_43 128M 2024-09-12T06:44:22.191 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:44:22.191 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:44:22.191 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:44:22.191 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:44:22.191 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_43 2024-09-12T06:44:22.191 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:44:22.192 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local size=128 2024-09-12T06:44:22.192 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n 128M ']' 2024-09-12T06:44:22.192 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ size=128M 2024-09-12T06:44:22.192 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:44:22.192 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_43 2024-09-12T06:44:22.320 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T06:44:22.320 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:44:22.320 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' pool = image ']' 2024-09-12T06:44:22.320 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ case "${pool}" in 2024-09-12T06:44:22.320 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 1 2024-09-12T06:44:22.320 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n '' ']' 2024-09-12T06:44:22.320 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ stress_write_image cluster2 mirror image_43 2024-09-12T06:44:22.320 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:44:22.321 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:44:22.321 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_43 2024-09-12T06:44:22.321 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T06:44:22.321 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local duration=9 2024-09-12T06:44:22.321 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set +e 2024-09-12T06:44:22.321 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_43 --debug-rbd=20 --debug-journaler=20 2024-09-12T06:44:23.209 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:100 IOs, 10923008 bytes 2024-09-12T06:44:25.482 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:200 IOs, 38470656 bytes 2024-09-12T06:44:27.344 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:300 IOs, 73553920 bytes 2024-09-12T06:44:28.338 INFO:tasks.ceph.cluster2.osd.0.smithi152.stderr:2024-09-12T06:44:28.340+0000 7fd61689f640 -1 reset not still connected to 0x55e76b50edd0 2024-09-12T06:44:28.342 INFO:tasks.ceph.cluster2.osd.0.smithi152.stderr:2024-09-12T06:44:28.343+0000 7fd61689f640 -1 reset not still connected to 0x55e7772cba00 2024-09-12T06:44:28.342 INFO:tasks.ceph.cluster2.osd.0.smithi152.stderr:2024-09-12T06:44:28.343+0000 7fd61689f640 -1 reset not still connected to 0x55e77b1fcc30 2024-09-12T06:44:28.342 INFO:tasks.ceph.cluster2.osd.0.smithi152.stderr:2024-09-12T06:44:28.343+0000 7fd61689f640 -1 reset not still connected to 0x55e77b1fcd00 2024-09-12T06:44:28.342 INFO:tasks.ceph.cluster2.osd.0.smithi152.stderr:2024-09-12T06:44:28.343+0000 7fd61689f640 -1 reset not still connected to 0x55e77c768680 2024-09-12T06:44:28.342 INFO:tasks.ceph.cluster2.osd.0.smithi152.stderr:2024-09-12T06:44:28.343+0000 7fd61689f640 -1 reset not still connected to 0x55e77d6f2d00 2024-09-12T06:44:28.342 INFO:tasks.ceph.cluster2.osd.0.smithi152.stderr:2024-09-12T06:44:28.343+0000 7fd61689f640 -1 reset not still connected to 0x55e77d6f3a00 2024-09-12T06:44:28.342 INFO:tasks.ceph.cluster2.osd.0.smithi152.stderr:2024-09-12T06:44:28.343+0000 7fd61689f640 -1 reset not still connected to 0x55e782f70750 2024-09-12T06:44:28.357 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:28.358+0000 7fa325ae1640 -1 reset not still connected to 0x55f85cc67040 2024-09-12T06:44:28.357 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:28.358+0000 7fa325ae1640 -1 reset not still connected to 0x55f869fdc0d0 2024-09-12T06:44:28.357 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:28.358+0000 7fa325ae1640 -1 reset not still connected to 0x55f86ee2fe10 2024-09-12T06:44:28.359 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:28.360+0000 7fa325ae1640 -1 reset not still connected to 0x55f88041e270 2024-09-12T06:44:28.362 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:28.364+0000 7fa325ae1640 -1 reset not still connected to 0x55f884309c70 2024-09-12T06:44:28.363 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:28.364+0000 7fa325ae1640 -1 reset not still connected to 0x55f8849a1040 2024-09-12T06:44:28.363 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:28.364+0000 7fa325ae1640 -1 reset not still connected to 0x55f8849a11e0 2024-09-12T06:44:28.363 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:28.364+0000 7fa325ae1640 -1 reset not still connected to 0x55f8863b4340 2024-09-12T06:44:28.363 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:28.364+0000 7fa325ae1640 -1 reset not still connected to 0x55f8863b4680 2024-09-12T06:44:28.365 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:28.367+0000 7fa325ae1640 -1 reset not still connected to 0x55f8863b5790 2024-09-12T06:44:28.366 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:28.367+0000 7fa325ae1640 -1 reset not still connected to 0x55f88b2cd5f0 2024-09-12T06:44:28.366 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:28.367+0000 7fa325ae1640 -1 reset not still connected to 0x55f890e69a00 2024-09-12T06:44:28.366 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:28.367+0000 7fa325ae1640 -1 reset not still connected to 0x55f8926ce0d0 2024-09-12T06:44:28.366 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:28.367+0000 7fa325ae1640 -1 reset not still connected to 0x55f896243040 2024-09-12T06:44:28.538 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:400 IOs, 89291776 bytes 2024-09-12T06:44:29.140 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:500 IOs, 105229312 bytes 2024-09-12T06:44:29.660 INFO:tasks.ceph.cluster2.osd.1.smithi152.stderr:2024-09-12T06:44:29.660+0000 7f33f9c90640 -1 reset not still connected to 0x55c758d9fd40 2024-09-12T06:44:30.122 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:30.124+0000 7fac12a52640 -1 reset not still connected to 0x55749aea7790 2024-09-12T06:44:30.122 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:30.124+0000 7fac12a52640 -1 reset not still connected to 0x55749b1cbc70 2024-09-12T06:44:30.221 INFO:tasks.ceph.cluster2.osd.1.smithi152.stderr:2024-09-12T06:44:30.223+0000 7f33f9c90640 -1 reset not still connected to 0x55c756b292b0 2024-09-12T06:44:30.772 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:600 IOs, 143067136 bytes 2024-09-12T06:44:31.331 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ error_code=124 2024-09-12T06:44:31.331 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set -e 2024-09-12T06:44:31.331 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T06:44:31.331 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:44:31.331 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:44:31.331 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_44 2024-09-12T06:44:31.331 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image_and_enable_mirror cluster2 mirror image_44 journal 128M 2024-09-12T06:44:31.331 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:44:31.332 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:44:31.332 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:44:31.332 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:44:31.332 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_44 2024-09-12T06:44:31.332 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:44:31.332 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local mode=journal 2024-09-12T06:44:31.332 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n journal ']' 2024-09-12T06:44:31.332 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:44:31.332 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image cluster2 mirror image_44 128M 2024-09-12T06:44:31.332 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:44:31.332 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:44:31.332 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:44:31.332 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:44:31.332 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_44 2024-09-12T06:44:31.333 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:44:31.333 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local size=128 2024-09-12T06:44:31.333 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n 128M ']' 2024-09-12T06:44:31.333 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ size=128M 2024-09-12T06:44:31.333 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:44:31.333 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_44 2024-09-12T06:44:31.429 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T06:44:31.429 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:44:31.429 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' pool = image ']' 2024-09-12T06:44:31.429 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ case "${pool}" in 2024-09-12T06:44:31.429 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 1 2024-09-12T06:44:31.429 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n '' ']' 2024-09-12T06:44:31.429 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ stress_write_image cluster2 mirror image_44 2024-09-12T06:44:31.429 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:44:31.429 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:44:31.429 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_44 2024-09-12T06:44:31.429 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T06:44:31.430 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local duration=9 2024-09-12T06:44:31.430 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set +e 2024-09-12T06:44:31.430 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_44 --debug-rbd=20 --debug-journaler=20 2024-09-12T06:44:32.755 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:32.754+0000 7fac12a52640 -1 reset not still connected to 0x55749b1cb860 2024-09-12T06:44:32.755 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:32.754+0000 7fac12a52640 -1 reset not still connected to 0x55749b1cbba0 2024-09-12T06:44:32.755 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:32.754+0000 7fac12a52640 -1 reset not still connected to 0x55749b2bf110 2024-09-12T06:44:32.755 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:32.756+0000 7fac12a52640 -1 reset not still connected to 0x55749b2bfee0 2024-09-12T06:44:32.755 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:32.756+0000 7fac12a52640 -1 reset not still connected to 0x55749c9e9380 2024-09-12T06:44:32.757 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:32.759+0000 7fac12a52640 -1 reset not still connected to 0x55749d3c4dd0 2024-09-12T06:44:32.757 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:32.759+0000 7fac12a52640 -1 reset not still connected to 0x55749d402410 2024-09-12T06:44:32.757 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:32.759+0000 7fac12a52640 -1 reset not still connected to 0x55749d402680 2024-09-12T06:44:32.758 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:32.759+0000 7fac12a52640 -1 reset not still connected to 0x55749d403ba0 2024-09-12T06:44:32.758 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:32.759+0000 7fac12a52640 -1 reset not still connected to 0x55749d7ca270 2024-09-12T06:44:32.760 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:32.762+0000 7fac12a52640 -1 reset not still connected to 0x55749d7ca8f0 2024-09-12T06:44:32.765 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:32.766+0000 7fac12a52640 -1 reset not still connected to 0x55749e05cf70 2024-09-12T06:44:32.765 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:32.766+0000 7fac12a52640 -1 reset not still connected to 0x5574a2e61d40 2024-09-12T06:44:32.765 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:32.766+0000 7fac12a52640 -1 reset not still connected to 0x5574a2fa7a00 2024-09-12T06:44:32.765 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:32.766+0000 7fac12a52640 -1 reset not still connected to 0x5574a62b6d00 2024-09-12T06:44:32.765 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:32.766+0000 7fac12a52640 -1 reset not still connected to 0x5574bdabf110 2024-09-12T06:44:32.765 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:32.766+0000 7fac12a52640 -1 reset not still connected to 0x5574bdabf1e0 2024-09-12T06:44:32.765 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:32.766+0000 7fac12a52640 -1 reset not still connected to 0x5574bdabfd40 2024-09-12T06:44:32.766 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:32.766+0000 7fac12a52640 -1 reset not still connected to 0x5574cfee1c70 2024-09-12T06:44:32.766 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:32.766+0000 7fac12a52640 -1 reset not still connected to 0x5574d1d6da00 2024-09-12T06:44:32.766 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:32.766+0000 7fac12a52640 -1 reset not still connected to 0x5574d5072c30 2024-09-12T06:44:32.997 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:44:32.996+0000 7f80d15bb640 -1 reset not still connected to 0x556b32cfef70 2024-09-12T06:44:32.998 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:44:32.998+0000 7f80d15bb640 -1 reset not still connected to 0x556b3e8a9790 2024-09-12T06:44:33.359 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:33.359+0000 7fa325ae1640 -1 reset not still connected to 0x55f85ac224e0 2024-09-12T06:44:33.363 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:33.364+0000 7fa325ae1640 -1 reset not still connected to 0x55f85ac229c0 2024-09-12T06:44:33.364 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:33.364+0000 7fa325ae1640 -1 reset not still connected to 0x55f85ba925b0 2024-09-12T06:44:33.364 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:33.364+0000 7fa325ae1640 -1 reset not still connected to 0x55f85ba92d00 2024-09-12T06:44:33.364 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:33.364+0000 7fa325ae1640 -1 reset not still connected to 0x55f85bac29c0 2024-09-12T06:44:33.364 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:33.364+0000 7fa325ae1640 -1 reset not still connected to 0x55f85bac2b60 2024-09-12T06:44:33.364 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:33.365+0000 7fa325ae1640 -1 reset not still connected to 0x55f85cc67860 2024-09-12T06:44:33.364 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:33.365+0000 7fa325ae1640 -1 reset not still connected to 0x55f85cc67930 2024-09-12T06:44:33.364 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:33.365+0000 7fa325ae1640 -1 reset not still connected to 0x55f85d61d380 2024-09-12T06:44:33.364 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:33.365+0000 7fa325ae1640 -1 reset not still connected to 0x55f8636984e0 2024-09-12T06:44:33.364 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:33.365+0000 7fa325ae1640 -1 reset not still connected to 0x55f863699040 2024-09-12T06:44:33.364 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:33.365+0000 7fa325ae1640 -1 reset not still connected to 0x55f86ee2e340 2024-09-12T06:44:33.364 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:33.365+0000 7fa325ae1640 -1 reset not still connected to 0x55f87faff040 2024-09-12T06:44:33.365 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:33.365+0000 7fa325ae1640 -1 reset not still connected to 0x55f88c6bfa00 2024-09-12T06:44:34.595 INFO:tasks.ceph.cluster2.osd.2.smithi152.stderr:2024-09-12T06:44:34.595+0000 7f4789beb640 -1 reset not still connected to 0x56163e3b2ea0 2024-09-12T06:44:34.595 INFO:tasks.ceph.cluster2.osd.2.smithi152.stderr:2024-09-12T06:44:34.595+0000 7f4789beb640 -1 reset not still connected to 0x56163e3b3ba0 2024-09-12T06:44:34.595 INFO:tasks.ceph.cluster2.osd.2.smithi152.stderr:2024-09-12T06:44:34.595+0000 7f4789beb640 -1 reset not still connected to 0x56163f9b6a90 2024-09-12T06:44:34.595 INFO:tasks.ceph.cluster2.osd.2.smithi152.stderr:2024-09-12T06:44:34.595+0000 7f4789beb640 -1 reset not still connected to 0x561640fa21a0 2024-09-12T06:44:34.595 INFO:tasks.ceph.cluster2.osd.2.smithi152.stderr:2024-09-12T06:44:34.595+0000 7f4789beb640 -1 reset not still connected to 0x5616436b4410 2024-09-12T06:44:34.595 INFO:tasks.ceph.cluster2.osd.2.smithi152.stderr:2024-09-12T06:44:34.595+0000 7f4789beb640 -1 reset not still connected to 0x5616576bfba0 2024-09-12T06:44:34.595 INFO:tasks.ceph.cluster2.osd.2.smithi152.stderr:2024-09-12T06:44:34.595+0000 7f4789beb640 -1 reset not still connected to 0x561658359790 2024-09-12T06:44:35.856 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:100 IOs, 88914944 bytes 2024-09-12T06:44:36.332 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:36.330+0000 7fac12a52640 -1 reset not still connected to 0x55749b2bf6c0 2024-09-12T06:44:36.444 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:36.338+0000 7fac12a52640 -1 reset not still connected to 0x55749b2bfc70 2024-09-12T06:44:36.444 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:36.338+0000 7fac12a52640 -1 reset not still connected to 0x55749c9a7860 2024-09-12T06:44:36.445 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:36.338+0000 7fac12a52640 -1 reset not still connected to 0x55749d3c4750 2024-09-12T06:44:36.445 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:36.338+0000 7fac12a52640 -1 reset not still connected to 0x55749e3c4750 2024-09-12T06:44:36.445 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:36.338+0000 7fac12a52640 -1 reset not still connected to 0x55749e3c48f0 2024-09-12T06:44:36.445 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:36.338+0000 7fac12a52640 -1 reset not still connected to 0x55749e3c4d00 2024-09-12T06:44:36.445 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:36.338+0000 7fac12a52640 -1 reset not still connected to 0x5574b3849380 2024-09-12T06:44:36.445 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:36.338+0000 7fac12a52640 -1 reset not still connected to 0x5574b7fd3c70 2024-09-12T06:44:36.445 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:36.340+0000 7fac12a52640 -1 reset not still connected to 0x5574cfee0680 2024-09-12T06:44:37.499 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:200 IOs, 112492544 bytes 2024-09-12T06:44:38.108 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:38.109+0000 7fa325ae1640 -1 reset not still connected to 0x55f85bac2b60 2024-09-12T06:44:38.109 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:38.110+0000 7fa325ae1640 -1 reset not still connected to 0x55f85cc67860 2024-09-12T06:44:38.109 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:38.110+0000 7fa325ae1640 -1 reset not still connected to 0x55f85cc67930 2024-09-12T06:44:38.109 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:38.110+0000 7fa325ae1640 -1 reset not still connected to 0x55f85d61d380 2024-09-12T06:44:38.117 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:38.114+0000 7fa325ae1640 -1 reset not still connected to 0x55f8636984e0 2024-09-12T06:44:38.117 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:38.114+0000 7fa325ae1640 -1 reset not still connected to 0x55f863699040 2024-09-12T06:44:38.117 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:38.114+0000 7fa325ae1640 -1 reset not still connected to 0x55f86ee2e340 2024-09-12T06:44:38.117 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:38.114+0000 7fa325ae1640 -1 reset not still connected to 0x55f87faff040 2024-09-12T06:44:38.117 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:38.114+0000 7fa325ae1640 -1 reset not still connected to 0x55f88c6bfa00 2024-09-12T06:44:38.117 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:38.114+0000 7fa325ae1640 -1 reset not still connected to 0x55f8926ce750 2024-09-12T06:44:38.117 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:38.114+0000 7fa325ae1640 -1 reset not still connected to 0x55f8927168f0 2024-09-12T06:44:38.259 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:300 IOs, 129011712 bytes 2024-09-12T06:44:40.444 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ error_code=124 2024-09-12T06:44:40.444 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set -e 2024-09-12T06:44:40.444 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T06:44:40.445 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:44:40.445 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:44:40.445 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_45 2024-09-12T06:44:40.445 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image_and_enable_mirror cluster2 mirror image_45 journal 128M 2024-09-12T06:44:40.445 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:44:40.445 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:44:40.445 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:44:40.445 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:44:40.445 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_45 2024-09-12T06:44:40.445 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:44:40.445 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local mode=journal 2024-09-12T06:44:40.445 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n journal ']' 2024-09-12T06:44:40.446 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:44:40.446 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image cluster2 mirror image_45 128M 2024-09-12T06:44:40.446 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:44:40.446 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:44:40.446 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:44:40.446 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:44:40.446 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_45 2024-09-12T06:44:40.446 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:44:40.446 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local size=128 2024-09-12T06:44:40.446 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n 128M ']' 2024-09-12T06:44:40.446 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ size=128M 2024-09-12T06:44:40.446 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:44:40.446 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_45 2024-09-12T06:44:40.521 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T06:44:40.521 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:44:40.521 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' pool = image ']' 2024-09-12T06:44:40.521 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ case "${pool}" in 2024-09-12T06:44:40.521 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 1 2024-09-12T06:44:40.521 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n '' ']' 2024-09-12T06:44:40.522 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ stress_write_image cluster2 mirror image_45 2024-09-12T06:44:40.522 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:44:40.522 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:44:40.522 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_45 2024-09-12T06:44:40.522 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T06:44:40.522 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local duration=8 2024-09-12T06:44:40.522 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set +e 2024-09-12T06:44:40.522 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_45 --debug-rbd=20 --debug-journaler=20 2024-09-12T06:44:40.859 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:40.861+0000 7fa325ae1640 -1 reset not still connected to 0x55f88c6bfa00 2024-09-12T06:44:40.860 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:40.861+0000 7fa325ae1640 -1 reset not still connected to 0x55f8926ce750 2024-09-12T06:44:41.537 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:100 IOs, 12981248 bytes 2024-09-12T06:44:43.095 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:43.096+0000 7fac12a52640 -1 reset not still connected to 0x55749b2bf110 2024-09-12T06:44:43.096 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:43.096+0000 7fac12a52640 -1 reset not still connected to 0x55749b2bfee0 2024-09-12T06:44:43.103 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:43.102+0000 7fac12a52640 -1 reset not still connected to 0x55749d402410 2024-09-12T06:44:43.103 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:43.103+0000 7fac12a52640 -1 reset not still connected to 0x55749d402680 2024-09-12T06:44:43.106 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:43.106+0000 7fac12a52640 -1 reset not still connected to 0x55749d403ba0 2024-09-12T06:44:43.106 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:43.106+0000 7fac12a52640 -1 reset not still connected to 0x55749d7ca270 2024-09-12T06:44:43.107 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:43.109+0000 7fac12a52640 -1 reset not still connected to 0x55749d7ca8f0 2024-09-12T06:44:43.107 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:43.109+0000 7fac12a52640 -1 reset not still connected to 0x55749e05cf70 2024-09-12T06:44:43.115 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:43.117+0000 7fac12a52640 -1 reset not still connected to 0x5574a2e61d40 2024-09-12T06:44:43.117 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:43.118+0000 7fac12a52640 -1 reset not still connected to 0x5574a2fa7a00 2024-09-12T06:44:43.118 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:43.118+0000 7fac12a52640 -1 reset not still connected to 0x5574a62b6d00 2024-09-12T06:44:43.118 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:43.118+0000 7fac12a52640 -1 reset not still connected to 0x5574bdabf110 2024-09-12T06:44:43.118 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:43.118+0000 7fac12a52640 -1 reset not still connected to 0x5574bdabf1e0 2024-09-12T06:44:43.118 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:43.118+0000 7fac12a52640 -1 reset not still connected to 0x5574bdabfd40 2024-09-12T06:44:43.118 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:43.118+0000 7fac12a52640 -1 reset not still connected to 0x5574cfee1c70 2024-09-12T06:44:43.118 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:43.118+0000 7fac12a52640 -1 reset not still connected to 0x5574d1d6da00 2024-09-12T06:44:43.118 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:43.118+0000 7fac12a52640 -1 reset not still connected to 0x5574d5072c30 2024-09-12T06:44:43.593 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:43.595+0000 7fac12a52640 -1 reset not still connected to 0x55749b1cb860 2024-09-12T06:44:43.598 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:43.597+0000 7fac12a52640 -1 reset not still connected to 0x55749b1cbba0 2024-09-12T06:44:43.598 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:43.597+0000 7fac12a52640 -1 reset not still connected to 0x55749b2bf110 2024-09-12T06:44:43.599 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:43.597+0000 7fac12a52640 -1 reset not still connected to 0x55749b2bfee0 2024-09-12T06:44:43.599 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:43.597+0000 7fac12a52640 -1 reset not still connected to 0x55749c9e9380 2024-09-12T06:44:43.599 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:43.597+0000 7fac12a52640 -1 reset not still connected to 0x55749d3c4dd0 2024-09-12T06:44:43.599 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:43.597+0000 7fac12a52640 -1 reset not still connected to 0x55749d402410 2024-09-12T06:44:43.599 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:43.597+0000 7fac12a52640 -1 reset not still connected to 0x55749d402680 2024-09-12T06:44:43.599 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:43.597+0000 7fac12a52640 -1 reset not still connected to 0x55749d403ba0 2024-09-12T06:44:43.599 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:43.597+0000 7fac12a52640 -1 reset not still connected to 0x55749d7ca270 2024-09-12T06:44:43.599 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:43.597+0000 7fac12a52640 -1 reset not still connected to 0x55749d7ca8f0 2024-09-12T06:44:43.599 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:43.597+0000 7fac12a52640 -1 reset not still connected to 0x55749e05cf70 2024-09-12T06:44:43.599 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:43.597+0000 7fac12a52640 -1 reset not still connected to 0x5574a2e61d40 2024-09-12T06:44:43.599 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:43.597+0000 7fac12a52640 -1 reset not still connected to 0x5574a2fa7a00 2024-09-12T06:44:43.599 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:43.597+0000 7fac12a52640 -1 reset not still connected to 0x5574a62b6d00 2024-09-12T06:44:43.599 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:43.597+0000 7fac12a52640 -1 reset not still connected to 0x5574bdabf110 2024-09-12T06:44:43.604 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:43.606+0000 7fac12a52640 -1 reset not still connected to 0x5574bdabf1e0 2024-09-12T06:44:43.604 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:43.606+0000 7fac12a52640 -1 reset not still connected to 0x5574bdabfd40 2024-09-12T06:44:43.605 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:43.606+0000 7fac12a52640 -1 reset not still connected to 0x5574cfee1c70 2024-09-12T06:44:43.605 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:43.606+0000 7fac12a52640 -1 reset not still connected to 0x5574d1d6da00 2024-09-12T06:44:43.607 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:43.607+0000 7fac12a52640 -1 reset not still connected to 0x5574d5072c30 2024-09-12T06:44:44.832 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:44.833+0000 7fac12a52640 -1 reset not still connected to 0x55749b2bfee0 2024-09-12T06:44:44.836 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:44.837+0000 7fac12a52640 -1 reset not still connected to 0x55749d3c4dd0 2024-09-12T06:44:44.840 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:44.840+0000 7fac12a52640 -1 reset not still connected to 0x55749d402410 2024-09-12T06:44:44.840 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:44.840+0000 7fac12a52640 -1 reset not still connected to 0x55749d402680 2024-09-12T06:44:44.840 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:44.840+0000 7fac12a52640 -1 reset not still connected to 0x55749d403ba0 2024-09-12T06:44:44.840 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:44.840+0000 7fac12a52640 -1 reset not still connected to 0x55749d7ca270 2024-09-12T06:44:44.840 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:44.840+0000 7fac12a52640 -1 reset not still connected to 0x55749e05cf70 2024-09-12T06:44:44.846 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:44.848+0000 7fac12a52640 -1 reset not still connected to 0x5574a2e61d40 2024-09-12T06:44:44.847 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:44.848+0000 7fac12a52640 -1 reset not still connected to 0x5574a2fa7a00 2024-09-12T06:44:44.847 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:44.848+0000 7fac12a52640 -1 reset not still connected to 0x5574a62b6d00 2024-09-12T06:44:44.847 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:44.848+0000 7fac12a52640 -1 reset not still connected to 0x5574bdabf110 2024-09-12T06:44:44.847 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:44.848+0000 7fac12a52640 -1 reset not still connected to 0x5574bdabf1e0 2024-09-12T06:44:44.847 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:44.848+0000 7fac12a52640 -1 reset not still connected to 0x5574bdabfd40 2024-09-12T06:44:44.847 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:44.848+0000 7fac12a52640 -1 reset not still connected to 0x5574cfee1c70 2024-09-12T06:44:44.849 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:44.850+0000 7fac12a52640 -1 reset not still connected to 0x5574d1d6da00 2024-09-12T06:44:44.849 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:44.850+0000 7fac12a52640 -1 reset not still connected to 0x5574d5072c30 2024-09-12T06:44:45.804 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:200 IOs, 93180928 bytes 2024-09-12T06:44:45.981 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:44:45.981+0000 7f80d15bb640 -1 reset not still connected to 0x556b03866f70 2024-09-12T06:44:45.981 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:44:45.981+0000 7f80d15bb640 -1 reset not still connected to 0x556b04ee6410 2024-09-12T06:44:45.981 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:44:45.981+0000 7f80d15bb640 -1 reset not still connected to 0x556b04ee6b60 2024-09-12T06:44:45.981 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:44:45.981+0000 7f80d15bb640 -1 reset not still connected to 0x556b07468000 2024-09-12T06:44:45.981 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:44:45.981+0000 7f80d15bb640 -1 reset not still connected to 0x556b0bcea9c0 2024-09-12T06:44:45.981 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:44:45.981+0000 7f80d15bb640 -1 reset not still connected to 0x556b33ff76c0 2024-09-12T06:44:46.278 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:46.279+0000 7fac12a52640 -1 reset not still connected to 0x55749b2bfee0 2024-09-12T06:44:46.278 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:46.279+0000 7fac12a52640 -1 reset not still connected to 0x55749d3c4dd0 2024-09-12T06:44:46.278 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:46.279+0000 7fac12a52640 -1 reset not still connected to 0x55749d402410 2024-09-12T06:44:46.279 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:46.279+0000 7fac12a52640 -1 reset not still connected to 0x55749d402680 2024-09-12T06:44:46.279 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:46.279+0000 7fac12a52640 -1 reset not still connected to 0x55749d403ba0 2024-09-12T06:44:46.279 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:46.279+0000 7fac12a52640 -1 reset not still connected to 0x55749d7ca270 2024-09-12T06:44:46.280 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:46.281+0000 7fac12a52640 -1 reset not still connected to 0x5574a2e61d40 2024-09-12T06:44:46.280 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:46.281+0000 7fac12a52640 -1 reset not still connected to 0x5574a2fa7a00 2024-09-12T06:44:46.280 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:46.281+0000 7fac12a52640 -1 reset not still connected to 0x5574a62b6d00 2024-09-12T06:44:46.280 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:46.281+0000 7fac12a52640 -1 reset not still connected to 0x5574bdabf110 2024-09-12T06:44:46.280 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:46.281+0000 7fac12a52640 -1 reset not still connected to 0x5574bdabf1e0 2024-09-12T06:44:46.280 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:46.282+0000 7fac12a52640 -1 reset not still connected to 0x5574cfee1c70 2024-09-12T06:44:46.281 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:46.282+0000 7fac12a52640 -1 reset not still connected to 0x5574d5072c30 2024-09-12T06:44:46.397 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:46.398+0000 7fac12a52640 -1 reset not still connected to 0x55749b2bfee0 2024-09-12T06:44:46.397 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:46.398+0000 7fac12a52640 -1 reset not still connected to 0x55749c9e9380 2024-09-12T06:44:46.400 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:46.400+0000 7fac12a52640 -1 reset not still connected to 0x55749d3c4dd0 2024-09-12T06:44:46.403 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:46.404+0000 7fac12a52640 -1 reset not still connected to 0x55749d402410 2024-09-12T06:44:46.403 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:46.404+0000 7fac12a52640 -1 reset not still connected to 0x55749d402680 2024-09-12T06:44:46.403 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:46.404+0000 7fac12a52640 -1 reset not still connected to 0x55749d403ba0 2024-09-12T06:44:46.403 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:46.404+0000 7fac12a52640 -1 reset not still connected to 0x55749d7ca270 2024-09-12T06:44:46.404 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:46.405+0000 7fac12a52640 -1 reset not still connected to 0x55749d7ca8f0 2024-09-12T06:44:46.404 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:46.405+0000 7fac12a52640 -1 reset not still connected to 0x55749e05cf70 2024-09-12T06:44:46.404 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:46.405+0000 7fac12a52640 -1 reset not still connected to 0x5574a2fa7a00 2024-09-12T06:44:46.404 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:46.405+0000 7fac12a52640 -1 reset not still connected to 0x5574a62b6d00 2024-09-12T06:44:46.406 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:46.408+0000 7fac12a52640 -1 reset not still connected to 0x5574bdabf1e0 2024-09-12T06:44:46.406 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:46.408+0000 7fac12a52640 -1 reset not still connected to 0x5574d1d6da00 2024-09-12T06:44:46.406 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:46.408+0000 7fac12a52640 -1 reset not still connected to 0x5574d5072c30 2024-09-12T06:44:47.096 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:47.098+0000 7fa325ae1640 -1 reset not still connected to 0x55f86ee2fe10 2024-09-12T06:44:47.101 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:47.102+0000 7fa325ae1640 -1 reset not still connected to 0x55f880b4e4e0 2024-09-12T06:44:47.101 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:47.102+0000 7fa325ae1640 -1 reset not still connected to 0x55f88417c410 2024-09-12T06:44:47.102 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:47.102+0000 7fa325ae1640 -1 reset not still connected to 0x55f884309c70 2024-09-12T06:44:47.102 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:47.103+0000 7fa325ae1640 -1 reset not still connected to 0x55f8849a1040 2024-09-12T06:44:47.103 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:47.103+0000 7fa325ae1640 -1 reset not still connected to 0x55f8849a11e0 2024-09-12T06:44:47.103 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:47.103+0000 7fa325ae1640 -1 reset not still connected to 0x55f8863b4340 2024-09-12T06:44:47.103 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:47.103+0000 7fa325ae1640 -1 reset not still connected to 0x55f8863b4680 2024-09-12T06:44:47.103 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:47.103+0000 7fa325ae1640 -1 reset not still connected to 0x55f8863b5790 2024-09-12T06:44:47.103 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:47.103+0000 7fa325ae1640 -1 reset not still connected to 0x55f88b2cd5f0 2024-09-12T06:44:47.103 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:47.103+0000 7fa325ae1640 -1 reset not still connected to 0x55f890e69a00 2024-09-12T06:44:47.104 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:47.106+0000 7fa325ae1640 -1 reset not still connected to 0x55f8926ce0d0 2024-09-12T06:44:47.104 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:47.106+0000 7fa325ae1640 -1 reset not still connected to 0x55f896243040 2024-09-12T06:44:47.638 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:300 IOs, 113116160 bytes 2024-09-12T06:44:48.038 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:48.037+0000 7fa325ae1640 -1 reset not still connected to 0x55f86ee2fe10 2024-09-12T06:44:48.038 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:48.037+0000 7fa325ae1640 -1 reset not still connected to 0x55f88041e270 2024-09-12T06:44:48.039 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:48.039+0000 7fa325ae1640 -1 reset not still connected to 0x55f880b4e4e0 2024-09-12T06:44:48.039 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:48.039+0000 7fa325ae1640 -1 reset not still connected to 0x55f88417c410 2024-09-12T06:44:48.039 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:48.039+0000 7fa325ae1640 -1 reset not still connected to 0x55f884309c70 2024-09-12T06:44:48.039 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:48.039+0000 7fa325ae1640 -1 reset not still connected to 0x55f8849a1040 2024-09-12T06:44:48.039 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:48.039+0000 7fa325ae1640 -1 reset not still connected to 0x55f8849a11e0 2024-09-12T06:44:48.039 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:48.039+0000 7fa325ae1640 -1 reset not still connected to 0x55f8863b4340 2024-09-12T06:44:48.041 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:48.042+0000 7fa325ae1640 -1 reset not still connected to 0x55f8863b5790 2024-09-12T06:44:48.041 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:48.042+0000 7fa325ae1640 -1 reset not still connected to 0x55f88b2cd5f0 2024-09-12T06:44:48.041 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:48.042+0000 7fa325ae1640 -1 reset not still connected to 0x55f8926ce0d0 2024-09-12T06:44:48.041 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:48.042+0000 7fa325ae1640 -1 reset not still connected to 0x55f896243040 2024-09-12T06:44:48.537 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ error_code=124 2024-09-12T06:44:48.537 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set -e 2024-09-12T06:44:48.537 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T06:44:48.538 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:44:48.538 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:44:48.538 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_46 2024-09-12T06:44:48.538 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image_and_enable_mirror cluster2 mirror image_46 journal 128M 2024-09-12T06:44:48.538 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:44:48.538 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:44:48.538 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:44:48.538 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:44:48.538 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_46 2024-09-12T06:44:48.538 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:44:48.538 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local mode=journal 2024-09-12T06:44:48.538 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n journal ']' 2024-09-12T06:44:48.538 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:44:48.539 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image cluster2 mirror image_46 128M 2024-09-12T06:44:48.539 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:44:48.539 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:44:48.539 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:44:48.539 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:44:48.539 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_46 2024-09-12T06:44:48.539 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:44:48.539 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local size=128 2024-09-12T06:44:48.539 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n 128M ']' 2024-09-12T06:44:48.539 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ size=128M 2024-09-12T06:44:48.539 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:44:48.539 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_46 2024-09-12T06:44:48.601 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T06:44:48.601 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:44:48.601 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' pool = image ']' 2024-09-12T06:44:48.601 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ case "${pool}" in 2024-09-12T06:44:48.601 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 1 2024-09-12T06:44:48.602 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n '' ']' 2024-09-12T06:44:48.602 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ stress_write_image cluster2 mirror image_46 2024-09-12T06:44:48.602 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:44:48.602 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:44:48.602 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_46 2024-09-12T06:44:48.602 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T06:44:48.602 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local duration=13 2024-09-12T06:44:48.603 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set +e 2024-09-12T06:44:48.603 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_46 --debug-rbd=20 --debug-journaler=20 2024-09-12T06:44:48.793 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:48.792+0000 7fac12a52640 -1 reset not still connected to 0x55749aea6410 2024-09-12T06:44:48.793 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:48.792+0000 7fac12a52640 -1 reset not still connected to 0x55749b2bfee0 2024-09-12T06:44:48.793 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:48.792+0000 7fac12a52640 -1 reset not still connected to 0x55749d403ba0 2024-09-12T06:44:48.793 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:48.792+0000 7fac12a52640 -1 reset not still connected to 0x5574a2fa7a00 2024-09-12T06:44:48.793 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:48.792+0000 7fac12a52640 -1 reset not still connected to 0x5574a62b6d00 2024-09-12T06:44:48.793 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:44:48.792+0000 7fac12a52640 -1 reset not still connected to 0x5574bdabf1e0 2024-09-12T06:44:49.798 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:44:49.799+0000 7f80d15bb640 -1 reset not still connected to 0x556b043adba0 2024-09-12T06:44:49.803 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:44:49.804+0000 7f80d15bb640 -1 reset not still connected to 0x556b0c2c6750 2024-09-12T06:44:49.803 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:44:49.804+0000 7f80d15bb640 -1 reset not still connected to 0x556b14db6a90 2024-09-12T06:44:49.803 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:44:49.804+0000 7f80d15bb640 -1 reset not still connected to 0x556b32cfef70 2024-09-12T06:44:49.803 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:44:49.804+0000 7f80d15bb640 -1 reset not still connected to 0x556b3e8a9790 2024-09-12T06:44:51.304 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:44:51.302+0000 7f80d15bb640 -1 reset not still connected to 0x556b3e8a9790 2024-09-12T06:44:55.269 INFO:tasks.ceph.cluster2.osd.1.smithi152.stderr:2024-09-12T06:44:55.269+0000 7f33f9c90640 -1 reset not still connected to 0x55c7432b5a00 2024-09-12T06:44:55.271 INFO:tasks.ceph.cluster2.osd.1.smithi152.stderr:2024-09-12T06:44:55.273+0000 7f33f9c90640 -1 reset not still connected to 0x55c7435f2000 2024-09-12T06:44:55.271 INFO:tasks.ceph.cluster2.osd.1.smithi152.stderr:2024-09-12T06:44:55.273+0000 7f33f9c90640 -1 reset not still connected to 0x55c750335ad0 2024-09-12T06:44:55.503 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:55.505+0000 7fa325ae1640 -1 reset not still connected to 0x55f85a4fcdd0 2024-09-12T06:44:55.508 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:55.508+0000 7fa325ae1640 -1 reset not still connected to 0x55f85bac3520 2024-09-12T06:44:55.510 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:55.508+0000 7fa325ae1640 -1 reset not still connected to 0x55f85cc67040 2024-09-12T06:44:55.510 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:55.508+0000 7fa325ae1640 -1 reset not still connected to 0x55f863611110 2024-09-12T06:44:55.510 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:55.508+0000 7fa325ae1640 -1 reset not still connected to 0x55f869fdc0d0 2024-09-12T06:44:55.510 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:55.508+0000 7fa325ae1640 -1 reset not still connected to 0x55f86ee2fe10 2024-09-12T06:44:55.510 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:55.508+0000 7fa325ae1640 -1 reset not still connected to 0x55f88041e270 2024-09-12T06:44:55.511 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:55.510+0000 7fa325ae1640 -1 reset not still connected to 0x55f880b4e4e0 2024-09-12T06:44:55.511 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:55.510+0000 7fa325ae1640 -1 reset not still connected to 0x55f88417c410 2024-09-12T06:44:55.511 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:55.510+0000 7fa325ae1640 -1 reset not still connected to 0x55f884309c70 2024-09-12T06:44:55.511 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:55.510+0000 7fa325ae1640 -1 reset not still connected to 0x55f8849a1040 2024-09-12T06:44:55.511 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:55.511+0000 7fa325ae1640 -1 reset not still connected to 0x55f8849a11e0 2024-09-12T06:44:55.511 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:55.511+0000 7fa325ae1640 -1 reset not still connected to 0x55f8863b4340 2024-09-12T06:44:55.516 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:55.516+0000 7fa325ae1640 -1 reset not still connected to 0x55f8863b5790 2024-09-12T06:44:55.516 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:55.516+0000 7fa325ae1640 -1 reset not still connected to 0x55f88b2cd5f0 2024-09-12T06:44:55.516 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:55.516+0000 7fa325ae1640 -1 reset not still connected to 0x55f890e69a00 2024-09-12T06:44:55.518 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:55.519+0000 7fa325ae1640 -1 reset not still connected to 0x55f8926ce0d0 2024-09-12T06:44:55.518 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:44:55.519+0000 7fa325ae1640 -1 reset not still connected to 0x55f896243040 2024-09-12T06:44:56.379 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:100 IOs, 110406656 bytes 2024-09-12T06:44:56.724 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:200 IOs, 124311552 bytes 2024-09-12T06:44:59.518 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:300 IOs, 169769984 bytes 2024-09-12T06:45:01.346 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:45:01.347+0000 7f80d15bb640 -1 reset not still connected to 0x556b32cfef70 2024-09-12T06:45:01.346 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:45:01.347+0000 7f80d15bb640 -1 reset not still connected to 0x556b3e8a9790 2024-09-12T06:45:01.618 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ error_code=124 2024-09-12T06:45:01.619 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set -e 2024-09-12T06:45:01.619 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T06:45:01.619 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:45:01.619 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:45:01.619 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_47 2024-09-12T06:45:01.619 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image_and_enable_mirror cluster2 mirror image_47 journal 128M 2024-09-12T06:45:01.619 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:45:01.619 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:45:01.619 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:45:01.619 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:45:01.619 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_47 2024-09-12T06:45:01.619 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:45:01.619 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local mode=journal 2024-09-12T06:45:01.619 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n journal ']' 2024-09-12T06:45:01.620 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:45:01.620 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image cluster2 mirror image_47 128M 2024-09-12T06:45:01.620 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:45:01.620 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:45:01.620 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:45:01.620 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:45:01.620 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_47 2024-09-12T06:45:01.620 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:45:01.620 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local size=128 2024-09-12T06:45:01.620 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n 128M ']' 2024-09-12T06:45:01.620 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ size=128M 2024-09-12T06:45:01.620 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:45:01.620 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_47 2024-09-12T06:45:01.815 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T06:45:01.815 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:45:01.815 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' pool = image ']' 2024-09-12T06:45:01.815 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ case "${pool}" in 2024-09-12T06:45:01.815 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 1 2024-09-12T06:45:01.815 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n '' ']' 2024-09-12T06:45:01.815 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ stress_write_image cluster2 mirror image_47 2024-09-12T06:45:01.815 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:45:01.815 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:45:01.816 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_47 2024-09-12T06:45:01.816 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T06:45:01.816 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local duration=7 2024-09-12T06:45:01.816 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set +e 2024-09-12T06:45:01.817 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_47 --debug-rbd=20 --debug-journaler=20 2024-09-12T06:45:02.096 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:02.097+0000 7fa325ae1640 -1 reset not still connected to 0x55f85a4fcdd0 2024-09-12T06:45:02.097 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:02.097+0000 7fa325ae1640 -1 reset not still connected to 0x55f85bac3520 2024-09-12T06:45:02.097 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:02.098+0000 7fa325ae1640 -1 reset not still connected to 0x55f85cc67040 2024-09-12T06:45:02.097 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:02.098+0000 7fa325ae1640 -1 reset not still connected to 0x55f863611110 2024-09-12T06:45:02.097 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:02.098+0000 7fa325ae1640 -1 reset not still connected to 0x55f869fdc0d0 2024-09-12T06:45:02.097 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:02.098+0000 7fa325ae1640 -1 reset not still connected to 0x55f86ee2fe10 2024-09-12T06:45:02.102 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:02.102+0000 7fa325ae1640 -1 reset not still connected to 0x55f88041e270 2024-09-12T06:45:02.102 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:02.102+0000 7fa325ae1640 -1 reset not still connected to 0x55f880b4e4e0 2024-09-12T06:45:02.103 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:02.102+0000 7fa325ae1640 -1 reset not still connected to 0x55f88417c410 2024-09-12T06:45:02.103 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:02.102+0000 7fa325ae1640 -1 reset not still connected to 0x55f884309c70 2024-09-12T06:45:02.103 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:02.102+0000 7fa325ae1640 -1 reset not still connected to 0x55f8849a1040 2024-09-12T06:45:02.103 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:02.102+0000 7fa325ae1640 -1 reset not still connected to 0x55f8849a11e0 2024-09-12T06:45:02.103 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:02.102+0000 7fa325ae1640 -1 reset not still connected to 0x55f8863b4340 2024-09-12T06:45:02.109 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:02.111+0000 7fa325ae1640 -1 reset not still connected to 0x55f8863b4680 2024-09-12T06:45:02.109 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:02.111+0000 7fa325ae1640 -1 reset not still connected to 0x55f8863b5790 2024-09-12T06:45:02.109 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:02.111+0000 7fa325ae1640 -1 reset not still connected to 0x55f88b2cd5f0 2024-09-12T06:45:02.110 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:02.111+0000 7fa325ae1640 -1 reset not still connected to 0x55f890e69a00 2024-09-12T06:45:02.110 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:02.111+0000 7fa325ae1640 -1 reset not still connected to 0x55f8926ce0d0 2024-09-12T06:45:02.110 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:02.111+0000 7fa325ae1640 -1 reset not still connected to 0x55f896243040 2024-09-12T06:45:02.347 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:100 IOs, 4886528 bytes 2024-09-12T06:45:04.860 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:04.860+0000 7fac12a52640 -1 reset not still connected to 0x55749d402410 2024-09-12T06:45:04.860 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:04.860+0000 7fac12a52640 -1 reset not still connected to 0x55749d402680 2024-09-12T06:45:04.860 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:04.860+0000 7fac12a52640 -1 reset not still connected to 0x55749d403ba0 2024-09-12T06:45:04.861 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:04.860+0000 7fac12a52640 -1 reset not still connected to 0x55749d7ca270 2024-09-12T06:45:04.865 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:04.864+0000 7fac12a52640 -1 reset not still connected to 0x5574a2e61d40 2024-09-12T06:45:04.865 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:04.864+0000 7fac12a52640 -1 reset not still connected to 0x5574a2fa7a00 2024-09-12T06:45:04.865 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:04.864+0000 7fac12a52640 -1 reset not still connected to 0x5574bdabf110 2024-09-12T06:45:04.865 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:04.864+0000 7fac12a52640 -1 reset not still connected to 0x5574bdabf1e0 2024-09-12T06:45:04.865 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:04.864+0000 7fac12a52640 -1 reset not still connected to 0x5574bdabfd40 2024-09-12T06:45:04.865 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:04.864+0000 7fac12a52640 -1 reset not still connected to 0x5574cf6f9ba0 2024-09-12T06:45:04.865 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:04.864+0000 7fac12a52640 -1 reset not still connected to 0x5574cfee1c70 2024-09-12T06:45:04.865 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:04.864+0000 7fac12a52640 -1 reset not still connected to 0x5574d5072c30 2024-09-12T06:45:04.979 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:04.980+0000 7fa325ae1640 -1 reset not still connected to 0x55f859178ea0 2024-09-12T06:45:04.979 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:04.980+0000 7fa325ae1640 -1 reset not still connected to 0x55f859179380 2024-09-12T06:45:04.979 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:04.980+0000 7fa325ae1640 -1 reset not still connected to 0x55f85a304a90 2024-09-12T06:45:04.979 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:04.980+0000 7fa325ae1640 -1 reset not still connected to 0x55f85ac224e0 2024-09-12T06:45:04.979 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:04.980+0000 7fa325ae1640 -1 reset not still connected to 0x55f85ac229c0 2024-09-12T06:45:04.980 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:04.980+0000 7fa325ae1640 -1 reset not still connected to 0x55f85ba925b0 2024-09-12T06:45:04.980 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:04.980+0000 7fa325ae1640 -1 reset not still connected to 0x55f85bac29c0 2024-09-12T06:45:04.980 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:04.980+0000 7fa325ae1640 -1 reset not still connected to 0x55f85bac2b60 2024-09-12T06:45:04.980 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:04.980+0000 7fa325ae1640 -1 reset not still connected to 0x55f85cc67860 2024-09-12T06:45:04.980 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:04.980+0000 7fa325ae1640 -1 reset not still connected to 0x55f85cc67930 2024-09-12T06:45:04.980 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:04.980+0000 7fa325ae1640 -1 reset not still connected to 0x55f85d61d380 2024-09-12T06:45:04.981 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:04.982+0000 7fa325ae1640 -1 reset not still connected to 0x55f863699040 2024-09-12T06:45:04.981 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:04.982+0000 7fa325ae1640 -1 reset not still connected to 0x55f86ee2e340 2024-09-12T06:45:04.981 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:04.982+0000 7fa325ae1640 -1 reset not still connected to 0x55f87faff040 2024-09-12T06:45:04.981 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:04.982+0000 7fa325ae1640 -1 reset not still connected to 0x55f88c6bfa00 2024-09-12T06:45:04.981 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:04.982+0000 7fa325ae1640 -1 reset not still connected to 0x55f8926ce750 2024-09-12T06:45:04.981 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:04.982+0000 7fa325ae1640 -1 reset not still connected to 0x55f8927168f0 2024-09-12T06:45:05.269 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:200 IOs, 48838656 bytes 2024-09-12T06:45:07.749 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:45:07.749+0000 7f80d15bb640 -1 reset not still connected to 0x556b021731e0 2024-09-12T06:45:07.749 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:45:07.749+0000 7f80d15bb640 -1 reset not still connected to 0x556b02ee8dd0 2024-09-12T06:45:07.749 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:45:07.749+0000 7f80d15bb640 -1 reset not still connected to 0x556b02ee9520 2024-09-12T06:45:07.749 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:45:07.749+0000 7f80d15bb640 -1 reset not still connected to 0x556b030a2c30 2024-09-12T06:45:07.749 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:45:07.749+0000 7f80d15bb640 -1 reset not still connected to 0x556b043ac680 2024-09-12T06:45:07.749 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:45:07.749+0000 7f80d15bb640 -1 reset not still connected to 0x556b043adba0 2024-09-12T06:45:07.749 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:45:07.749+0000 7f80d15bb640 -1 reset not still connected to 0x556b044000d0 2024-09-12T06:45:07.749 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:45:07.750+0000 7f80d15bb640 -1 reset not still connected to 0x556b0c2c6750 2024-09-12T06:45:07.749 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:45:07.750+0000 7f80d15bb640 -1 reset not still connected to 0x556b14db6a90 2024-09-12T06:45:07.749 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:45:07.750+0000 7f80d15bb640 -1 reset not still connected to 0x556b32cfef70 2024-09-12T06:45:07.749 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:45:07.750+0000 7f80d15bb640 -1 reset not still connected to 0x556b3e8a9790 2024-09-12T06:45:08.185 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:45:08.187+0000 7f80d15bb640 -1 reset not still connected to 0x556b021731e0 2024-09-12T06:45:08.185 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:45:08.187+0000 7f80d15bb640 -1 reset not still connected to 0x556b02ee8dd0 2024-09-12T06:45:08.185 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:45:08.187+0000 7f80d15bb640 -1 reset not still connected to 0x556b02ee9520 2024-09-12T06:45:08.185 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:45:08.187+0000 7f80d15bb640 -1 reset not still connected to 0x556b030a2c30 2024-09-12T06:45:08.189 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:45:08.191+0000 7f80d15bb640 -1 reset not still connected to 0x556b043ac680 2024-09-12T06:45:08.191 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:45:08.191+0000 7f80d15bb640 -1 reset not still connected to 0x556b043adba0 2024-09-12T06:45:08.191 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:45:08.191+0000 7f80d15bb640 -1 reset not still connected to 0x556b044000d0 2024-09-12T06:45:08.194 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:45:08.193+0000 7f80d15bb640 -1 reset not still connected to 0x556b0c2c6750 2024-09-12T06:45:08.194 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:45:08.193+0000 7f80d15bb640 -1 reset not still connected to 0x556b100aa680 2024-09-12T06:45:08.194 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:45:08.194+0000 7f80d15bb640 -1 reset not still connected to 0x556b14db6a90 2024-09-12T06:45:08.194 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:45:08.194+0000 7f80d15bb640 -1 reset not still connected to 0x556b32cfef70 2024-09-12T06:45:08.194 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:45:08.194+0000 7f80d15bb640 -1 reset not still connected to 0x556b3e8a9790 2024-09-12T06:45:08.827 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ error_code=124 2024-09-12T06:45:08.827 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set -e 2024-09-12T06:45:08.827 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T06:45:08.827 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:45:08.828 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:45:08.828 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_48 2024-09-12T06:45:08.828 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image_and_enable_mirror cluster2 mirror image_48 journal 128M 2024-09-12T06:45:08.828 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:45:08.828 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:45:08.828 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:45:08.828 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:45:08.828 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_48 2024-09-12T06:45:08.828 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:45:08.828 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local mode=journal 2024-09-12T06:45:08.828 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n journal ']' 2024-09-12T06:45:08.828 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:45:08.829 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image cluster2 mirror image_48 128M 2024-09-12T06:45:08.829 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:45:08.829 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:45:08.829 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:45:08.829 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:45:08.829 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_48 2024-09-12T06:45:08.829 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:45:08.829 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local size=128 2024-09-12T06:45:08.829 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n 128M ']' 2024-09-12T06:45:08.829 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ size=128M 2024-09-12T06:45:08.829 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:45:08.829 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_48 2024-09-12T06:45:08.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T06:45:08.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:45:08.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' pool = image ']' 2024-09-12T06:45:08.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ case "${pool}" in 2024-09-12T06:45:08.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 1 2024-09-12T06:45:08.907 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n '' ']' 2024-09-12T06:45:08.907 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ stress_write_image cluster2 mirror image_48 2024-09-12T06:45:08.907 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:45:08.907 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:45:08.907 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_48 2024-09-12T06:45:08.907 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T06:45:08.907 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local duration=11 2024-09-12T06:45:08.907 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set +e 2024-09-12T06:45:08.908 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_48 --debug-rbd=20 --debug-journaler=20 2024-09-12T06:45:09.644 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:09.645+0000 7fac12a52640 -1 reset not still connected to 0x55749d4025b0 2024-09-12T06:45:09.645 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:09.645+0000 7fac12a52640 -1 reset not still connected to 0x55749e3c4750 2024-09-12T06:45:09.645 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:09.645+0000 7fac12a52640 -1 reset not still connected to 0x55749e3c48f0 2024-09-12T06:45:09.645 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:09.645+0000 7fac12a52640 -1 reset not still connected to 0x55749e3c4d00 2024-09-12T06:45:09.645 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:09.645+0000 7fac12a52640 -1 reset not still connected to 0x5574b3849380 2024-09-12T06:45:11.438 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:100 IOs, 58045440 bytes 2024-09-12T06:45:13.331 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:13.333+0000 7fa325ae1640 -1 reset not still connected to 0x55f85bac29c0 2024-09-12T06:45:13.333 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:13.333+0000 7fa325ae1640 -1 reset not still connected to 0x55f85cc67860 2024-09-12T06:45:13.333 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:13.333+0000 7fa325ae1640 -1 reset not still connected to 0x55f85cc67930 2024-09-12T06:45:13.334 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:13.333+0000 7fa325ae1640 -1 reset not still connected to 0x55f85d61d380 2024-09-12T06:45:13.336 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:13.337+0000 7fa325ae1640 -1 reset not still connected to 0x55f863699040 2024-09-12T06:45:13.336 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:13.337+0000 7fa325ae1640 -1 reset not still connected to 0x55f86ee2e340 2024-09-12T06:45:13.336 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:13.337+0000 7fa325ae1640 -1 reset not still connected to 0x55f87faff040 2024-09-12T06:45:13.336 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:13.337+0000 7fa325ae1640 -1 reset not still connected to 0x55f88c6bfa00 2024-09-12T06:45:13.336 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:13.337+0000 7fa325ae1640 -1 reset not still connected to 0x55f8926ce750 2024-09-12T06:45:13.336 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:13.337+0000 7fa325ae1640 -1 reset not still connected to 0x55f8927168f0 2024-09-12T06:45:13.613 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:13.614+0000 7fa325ae1640 -1 reset not still connected to 0x55f85bac2b60 2024-09-12T06:45:13.618 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:13.618+0000 7fa325ae1640 -1 reset not still connected to 0x55f85cc67930 2024-09-12T06:45:13.620 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:13.618+0000 7fa325ae1640 -1 reset not still connected to 0x55f85d61d380 2024-09-12T06:45:13.620 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:13.618+0000 7fa325ae1640 -1 reset not still connected to 0x55f8636984e0 2024-09-12T06:45:13.621 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:13.623+0000 7fa325ae1640 -1 reset not still connected to 0x55f863699040 2024-09-12T06:45:13.621 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:13.623+0000 7fa325ae1640 -1 reset not still connected to 0x55f86ee2e340 2024-09-12T06:45:13.621 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:13.623+0000 7fa325ae1640 -1 reset not still connected to 0x55f87faff040 2024-09-12T06:45:13.621 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:13.623+0000 7fa325ae1640 -1 reset not still connected to 0x55f88c6bfa00 2024-09-12T06:45:13.622 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:13.623+0000 7fa325ae1640 -1 reset not still connected to 0x55f8926ce750 2024-09-12T06:45:13.622 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:13.623+0000 7fa325ae1640 -1 reset not still connected to 0x55f8927168f0 2024-09-12T06:45:14.285 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:14.277+0000 7fac12a52640 -1 reset not still connected to 0x55749b2bfee0 2024-09-12T06:45:14.286 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:14.281+0000 7fac12a52640 -1 reset not still connected to 0x5574a2fa7a00 2024-09-12T06:45:15.489 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:45:15.490+0000 7f80d15bb640 -1 reset not still connected to 0x556b044000d0 2024-09-12T06:45:15.489 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:45:15.490+0000 7f80d15bb640 -1 reset not still connected to 0x556b0c2c6750 2024-09-12T06:45:15.489 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:45:15.490+0000 7f80d15bb640 -1 reset not still connected to 0x556b100aa680 2024-09-12T06:45:15.490 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:45:15.490+0000 7f80d15bb640 -1 reset not still connected to 0x556b14db6a90 2024-09-12T06:45:15.490 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:45:15.490+0000 7f80d15bb640 -1 reset not still connected to 0x556b32cfef70 2024-09-12T06:45:15.490 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:45:15.490+0000 7f80d15bb640 -1 reset not still connected to 0x556b3e8a9790 2024-09-12T06:45:15.516 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:200 IOs, 99354624 bytes 2024-09-12T06:45:15.800 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:45:15.801+0000 7f80d15bb640 -1 reset not still connected to 0x556b044000d0 2024-09-12T06:45:15.801 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:45:15.803+0000 7f80d15bb640 -1 reset not still connected to 0x556b0c2c6750 2024-09-12T06:45:15.801 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:45:15.803+0000 7f80d15bb640 -1 reset not still connected to 0x556b14db6a90 2024-09-12T06:45:15.802 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:45:15.803+0000 7f80d15bb640 -1 reset not still connected to 0x556b32cfef70 2024-09-12T06:45:15.802 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:45:15.803+0000 7f80d15bb640 -1 reset not still connected to 0x556b3e8a9790 2024-09-12T06:45:16.200 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:45:16.201+0000 7f80d15bb640 -1 reset not still connected to 0x556b03866f70 2024-09-12T06:45:16.200 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:45:16.201+0000 7f80d15bb640 -1 reset not still connected to 0x556b04ee6410 2024-09-12T06:45:16.200 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:45:16.201+0000 7f80d15bb640 -1 reset not still connected to 0x556b04ee6b60 2024-09-12T06:45:16.201 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:45:16.201+0000 7f80d15bb640 -1 reset not still connected to 0x556b05b5e680 2024-09-12T06:45:16.201 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:45:16.201+0000 7f80d15bb640 -1 reset not still connected to 0x556b07468000 2024-09-12T06:45:16.201 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:45:16.201+0000 7f80d15bb640 -1 reset not still connected to 0x556b0bcea9c0 2024-09-12T06:45:16.201 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:45:16.201+0000 7f80d15bb640 -1 reset not still connected to 0x556b33ff76c0 2024-09-12T06:45:16.771 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:300 IOs, 146184192 bytes 2024-09-12T06:45:19.917 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ error_code=124 2024-09-12T06:45:19.917 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set -e 2024-09-12T06:45:19.917 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T06:45:19.917 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:45:19.918 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:45:19.918 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_49 2024-09-12T06:45:19.918 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image_and_enable_mirror cluster2 mirror image_49 journal 128M 2024-09-12T06:45:19.918 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:45:19.918 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:45:19.918 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:45:19.918 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:45:19.918 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_49 2024-09-12T06:45:19.918 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:45:19.918 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local mode=journal 2024-09-12T06:45:19.918 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n journal ']' 2024-09-12T06:45:19.918 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:45:19.918 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image cluster2 mirror image_49 128M 2024-09-12T06:45:19.919 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:45:19.919 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:45:19.919 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:45:19.919 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:45:19.919 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_49 2024-09-12T06:45:19.919 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:45:19.919 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local size=128 2024-09-12T06:45:19.919 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n 128M ']' 2024-09-12T06:45:19.919 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ size=128M 2024-09-12T06:45:19.919 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:45:19.919 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_49 2024-09-12T06:45:20.001 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T06:45:20.001 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:45:20.001 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' pool = image ']' 2024-09-12T06:45:20.001 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ case "${pool}" in 2024-09-12T06:45:20.001 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 1 2024-09-12T06:45:20.001 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n '' ']' 2024-09-12T06:45:20.001 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ stress_write_image cluster2 mirror image_49 2024-09-12T06:45:20.001 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:45:20.001 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:45:20.002 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_49 2024-09-12T06:45:20.002 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T06:45:20.002 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local duration=13 2024-09-12T06:45:20.002 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set +e 2024-09-12T06:45:20.003 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_49 --debug-rbd=20 --debug-journaler=20 2024-09-12T06:45:20.464 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:100 IOs, 3492864 bytes 2024-09-12T06:45:21.156 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:45:21.157+0000 7f80d15bb640 -1 reset not still connected to 0x556b044000d0 2024-09-12T06:45:21.160 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:45:21.161+0000 7f80d15bb640 -1 reset not still connected to 0x556b14db6a90 2024-09-12T06:45:21.160 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:45:21.161+0000 7f80d15bb640 -1 reset not still connected to 0x556b32cfef70 2024-09-12T06:45:21.160 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:45:21.161+0000 7f80d15bb640 -1 reset not still connected to 0x556b3e8a9790 2024-09-12T06:45:21.277 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:45:21.276+0000 7f80d15bb640 -1 reset not still connected to 0x556b044000d0 2024-09-12T06:45:21.278 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:45:21.277+0000 7f80d15bb640 -1 reset not still connected to 0x556b0c2c6750 2024-09-12T06:45:21.282 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:45:21.284+0000 7f80d15bb640 -1 reset not still connected to 0x556b14db6a90 2024-09-12T06:45:21.282 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:45:21.284+0000 7f80d15bb640 -1 reset not still connected to 0x556b32cfef70 2024-09-12T06:45:21.282 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:45:21.284+0000 7f80d15bb640 -1 reset not still connected to 0x556b3e8a9790 2024-09-12T06:45:21.832 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:21.834+0000 7fa325ae1640 -1 reset not still connected to 0x55f859178ea0 2024-09-12T06:45:21.835 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:21.836+0000 7fa325ae1640 -1 reset not still connected to 0x55f859179380 2024-09-12T06:45:21.836 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:21.837+0000 7fa325ae1640 -1 reset not still connected to 0x55f85a304a90 2024-09-12T06:45:21.838 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:21.837+0000 7fa325ae1640 -1 reset not still connected to 0x55f85ac224e0 2024-09-12T06:45:21.838 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:21.837+0000 7fa325ae1640 -1 reset not still connected to 0x55f85ac229c0 2024-09-12T06:45:21.838 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:21.837+0000 7fa325ae1640 -1 reset not still connected to 0x55f85ba925b0 2024-09-12T06:45:21.838 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:21.837+0000 7fa325ae1640 -1 reset not still connected to 0x55f85ba92d00 2024-09-12T06:45:21.838 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:21.837+0000 7fa325ae1640 -1 reset not still connected to 0x55f85bac29c0 2024-09-12T06:45:21.839 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:21.837+0000 7fa325ae1640 -1 reset not still connected to 0x55f85bac2b60 2024-09-12T06:45:21.839 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:21.837+0000 7fa325ae1640 -1 reset not still connected to 0x55f85cc67860 2024-09-12T06:45:21.839 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:21.837+0000 7fa325ae1640 -1 reset not still connected to 0x55f85cc67930 2024-09-12T06:45:21.839 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:21.837+0000 7fa325ae1640 -1 reset not still connected to 0x55f85d61d380 2024-09-12T06:45:21.839 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:21.837+0000 7fa325ae1640 -1 reset not still connected to 0x55f8636984e0 2024-09-12T06:45:21.839 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:21.837+0000 7fa325ae1640 -1 reset not still connected to 0x55f863699040 2024-09-12T06:45:21.839 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:21.837+0000 7fa325ae1640 -1 reset not still connected to 0x55f86ee2e340 2024-09-12T06:45:21.839 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:21.837+0000 7fa325ae1640 -1 reset not still connected to 0x55f87faff040 2024-09-12T06:45:21.839 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:21.837+0000 7fa325ae1640 -1 reset not still connected to 0x55f88c6bfa00 2024-09-12T06:45:21.839 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:21.837+0000 7fa325ae1640 -1 reset not still connected to 0x55f8926ce750 2024-09-12T06:45:21.909 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:45:21.911+0000 7f80d15bb640 -1 reset not still connected to 0x556b33ff76c0 2024-09-12T06:45:26.482 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:200 IOs, 73396224 bytes 2024-09-12T06:45:28.811 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:45:28.810+0000 7f80d15bb640 -1 reset not still connected to 0x556b04ee6b60 2024-09-12T06:45:28.813 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:45:28.810+0000 7f80d15bb640 -1 reset not still connected to 0x556b05b5e680 2024-09-12T06:45:28.814 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:45:28.816+0000 7f80d15bb640 -1 reset not still connected to 0x556b07468000 2024-09-12T06:45:28.814 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:45:28.816+0000 7f80d15bb640 -1 reset not still connected to 0x556b0bcea9c0 2024-09-12T06:45:28.814 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:45:28.816+0000 7f80d15bb640 -1 reset not still connected to 0x556b33ff76c0 2024-09-12T06:45:29.896 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:29.897+0000 7fac12a52640 -1 reset not still connected to 0x55749b2be410 2024-09-12T06:45:30.252 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:300 IOs, 141189120 bytes 2024-09-12T06:45:32.021 INFO:tasks.ceph.cluster2.osd.2.smithi152.stderr:2024-09-12T06:45:32.023+0000 7f4789beb640 -1 reset not still connected to 0x5616436b4410 2024-09-12T06:45:32.022 INFO:tasks.ceph.cluster2.osd.2.smithi152.stderr:2024-09-12T06:45:32.023+0000 7f4789beb640 -1 reset not still connected to 0x5616522f7110 2024-09-12T06:45:32.022 INFO:tasks.ceph.cluster2.osd.2.smithi152.stderr:2024-09-12T06:45:32.023+0000 7f4789beb640 -1 reset not still connected to 0x5616576bfba0 2024-09-12T06:45:32.227 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:32.228+0000 7fa325ae1640 -1 reset not still connected to 0x55f85a3bab60 2024-09-12T06:45:32.230 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:32.231+0000 7fa325ae1640 -1 reset not still connected to 0x55f85a4fc9c0 2024-09-12T06:45:32.236 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:32.237+0000 7fa325ae1640 -1 reset not still connected to 0x55f85a4fcd00 2024-09-12T06:45:32.236 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:32.237+0000 7fa325ae1640 -1 reset not still connected to 0x55f85a4fcdd0 2024-09-12T06:45:32.236 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:32.237+0000 7fa325ae1640 -1 reset not still connected to 0x55f85ac22750 2024-09-12T06:45:32.236 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:32.237+0000 7fa325ae1640 -1 reset not still connected to 0x55f85bac3520 2024-09-12T06:45:32.237 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:32.239+0000 7fa325ae1640 -1 reset not still connected to 0x55f85cc67040 2024-09-12T06:45:32.237 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:32.239+0000 7fa325ae1640 -1 reset not still connected to 0x55f863611110 2024-09-12T06:45:32.237 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:32.239+0000 7fa325ae1640 -1 reset not still connected to 0x55f869fdc0d0 2024-09-12T06:45:32.237 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:32.239+0000 7fa325ae1640 -1 reset not still connected to 0x55f86ee2fe10 2024-09-12T06:45:32.238 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:32.239+0000 7fa325ae1640 -1 reset not still connected to 0x55f878a54410 2024-09-12T06:45:32.246 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:32.247+0000 7fa325ae1640 -1 reset not still connected to 0x55f88041e270 2024-09-12T06:45:32.246 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:32.247+0000 7fa325ae1640 -1 reset not still connected to 0x55f880b4e4e0 2024-09-12T06:45:32.246 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:32.247+0000 7fa325ae1640 -1 reset not still connected to 0x55f88417c410 2024-09-12T06:45:32.246 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:32.247+0000 7fa325ae1640 -1 reset not still connected to 0x55f884309c70 2024-09-12T06:45:32.246 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:32.247+0000 7fa325ae1640 -1 reset not still connected to 0x55f8849a1040 2024-09-12T06:45:32.246 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:32.247+0000 7fa325ae1640 -1 reset not still connected to 0x55f8849a11e0 2024-09-12T06:45:32.246 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:32.247+0000 7fa325ae1640 -1 reset not still connected to 0x55f8863b4340 2024-09-12T06:45:32.246 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:32.247+0000 7fa325ae1640 -1 reset not still connected to 0x55f8863b4680 2024-09-12T06:45:32.246 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:32.247+0000 7fa325ae1640 -1 reset not still connected to 0x55f8863b5790 2024-09-12T06:45:32.247 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:32.247+0000 7fa325ae1640 -1 reset not still connected to 0x55f88b2cd5f0 2024-09-12T06:45:32.247 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:32.247+0000 7fa325ae1640 -1 reset not still connected to 0x55f890e69a00 2024-09-12T06:45:32.247 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:32.247+0000 7fa325ae1640 -1 reset not still connected to 0x55f8926ce0d0 2024-09-12T06:45:32.247 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:32.247+0000 7fa325ae1640 -1 reset not still connected to 0x55f8926ce1a0 2024-09-12T06:45:32.247 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:32.247+0000 7fa325ae1640 -1 reset not still connected to 0x55f896243040 2024-09-12T06:45:33.013 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ error_code=124 2024-09-12T06:45:33.013 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set -e 2024-09-12T06:45:33.013 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T06:45:33.013 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:45:33.013 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:45:33.013 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_50 2024-09-12T06:45:33.013 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image_and_enable_mirror cluster2 mirror image_50 journal 128M 2024-09-12T06:45:33.013 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:45:33.013 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:45:33.013 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:45:33.013 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:45:33.014 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_50 2024-09-12T06:45:33.014 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:45:33.014 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local mode=journal 2024-09-12T06:45:33.014 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n journal ']' 2024-09-12T06:45:33.014 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:45:33.014 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image cluster2 mirror image_50 128M 2024-09-12T06:45:33.014 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:45:33.014 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:45:33.014 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:45:33.014 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:45:33.014 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_50 2024-09-12T06:45:33.014 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:45:33.014 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local size=128 2024-09-12T06:45:33.014 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n 128M ']' 2024-09-12T06:45:33.014 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ size=128M 2024-09-12T06:45:33.015 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:45:33.015 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_50 2024-09-12T06:45:33.093 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T06:45:33.093 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:45:33.093 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' pool = image ']' 2024-09-12T06:45:33.093 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ case "${pool}" in 2024-09-12T06:45:33.093 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 1 2024-09-12T06:45:33.094 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n '' ']' 2024-09-12T06:45:33.094 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ stress_write_image cluster2 mirror image_50 2024-09-12T06:45:33.094 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:45:33.094 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:45:33.094 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_50 2024-09-12T06:45:33.094 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T06:45:33.094 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local duration=5 2024-09-12T06:45:33.094 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set +e 2024-09-12T06:45:33.094 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_50 --debug-rbd=20 --debug-journaler=20 2024-09-12T06:45:33.175 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:33.169+0000 7fac12a52640 -1 reset not still connected to 0x55749b2bfee0 2024-09-12T06:45:33.175 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:33.169+0000 7fac12a52640 -1 reset not still connected to 0x55749c9e9380 2024-09-12T06:45:33.176 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:33.172+0000 7fac12a52640 -1 reset not still connected to 0x55749d3c4dd0 2024-09-12T06:45:33.176 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:33.172+0000 7fac12a52640 -1 reset not still connected to 0x55749d402680 2024-09-12T06:45:33.176 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:33.172+0000 7fac12a52640 -1 reset not still connected to 0x55749d403ba0 2024-09-12T06:45:33.176 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:33.175+0000 7fac12a52640 -1 reset not still connected to 0x55749d7ca270 2024-09-12T06:45:33.177 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:33.178+0000 7fac12a52640 -1 reset not still connected to 0x55749d7ca8f0 2024-09-12T06:45:33.177 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:33.178+0000 7fac12a52640 -1 reset not still connected to 0x55749e05cf70 2024-09-12T06:45:33.179 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:33.180+0000 7fac12a52640 -1 reset not still connected to 0x5574a2e61d40 2024-09-12T06:45:33.179 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:33.180+0000 7fac12a52640 -1 reset not still connected to 0x5574a2fa7a00 2024-09-12T06:45:33.179 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:33.180+0000 7fac12a52640 -1 reset not still connected to 0x5574a62b6d00 2024-09-12T06:45:33.180 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:33.182+0000 7fac12a52640 -1 reset not still connected to 0x5574bdabf110 2024-09-12T06:45:33.181 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:33.182+0000 7fac12a52640 -1 reset not still connected to 0x5574bdabf1e0 2024-09-12T06:45:33.182 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:33.184+0000 7fac12a52640 -1 reset not still connected to 0x5574bdabfd40 2024-09-12T06:45:33.182 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:33.184+0000 7fac12a52640 -1 reset not still connected to 0x5574cf6f9ba0 2024-09-12T06:45:33.182 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:33.184+0000 7fac12a52640 -1 reset not still connected to 0x5574cfee1c70 2024-09-12T06:45:33.182 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:33.184+0000 7fac12a52640 -1 reset not still connected to 0x5574d1d6da00 2024-09-12T06:45:33.182 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:33.184+0000 7fac12a52640 -1 reset not still connected to 0x5574d5072c30 2024-09-12T06:45:33.524 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:33.523+0000 7fac12a52640 -1 reset not still connected to 0x5574b3849380 2024-09-12T06:45:33.528 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:33.530+0000 7fac12a52640 -1 reset not still connected to 0x5574cfee0680 2024-09-12T06:45:34.836 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:34.836+0000 7fac12a52640 -1 reset not still connected to 0x55749b2bf110 2024-09-12T06:45:34.840 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:34.841+0000 7fac12a52640 -1 reset not still connected to 0x55749b2bfee0 2024-09-12T06:45:34.840 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:34.841+0000 7fac12a52640 -1 reset not still connected to 0x55749c9e9380 2024-09-12T06:45:34.844 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:34.846+0000 7fac12a52640 -1 reset not still connected to 0x55749d3c4dd0 2024-09-12T06:45:34.848 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:34.849+0000 7fac12a52640 -1 reset not still connected to 0x55749d402410 2024-09-12T06:45:34.848 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:34.849+0000 7fac12a52640 -1 reset not still connected to 0x55749d402680 2024-09-12T06:45:34.848 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:34.849+0000 7fac12a52640 -1 reset not still connected to 0x55749d403ba0 2024-09-12T06:45:34.851 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:34.849+0000 7fac12a52640 -1 reset not still connected to 0x55749d7ca270 2024-09-12T06:45:34.852 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:34.849+0000 7fac12a52640 -1 reset not still connected to 0x55749d7ca8f0 2024-09-12T06:45:34.852 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:34.849+0000 7fac12a52640 -1 reset not still connected to 0x55749e05cf70 2024-09-12T06:45:34.853 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:34.853+0000 7fac12a52640 -1 reset not still connected to 0x5574a2e61d40 2024-09-12T06:45:34.853 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:34.853+0000 7fac12a52640 -1 reset not still connected to 0x5574a2fa7a00 2024-09-12T06:45:34.856 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:34.853+0000 7fac12a52640 -1 reset not still connected to 0x5574a62b6d00 2024-09-12T06:45:34.856 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:34.853+0000 7fac12a52640 -1 reset not still connected to 0x5574bdabf110 2024-09-12T06:45:34.856 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:34.853+0000 7fac12a52640 -1 reset not still connected to 0x5574bdabf1e0 2024-09-12T06:45:34.856 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:34.853+0000 7fac12a52640 -1 reset not still connected to 0x5574cf6f9ba0 2024-09-12T06:45:34.856 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:34.855+0000 7fac12a52640 -1 reset not still connected to 0x5574cfee1c70 2024-09-12T06:45:34.857 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:34.858+0000 7fac12a52640 -1 reset not still connected to 0x5574d1d6da00 2024-09-12T06:45:34.857 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:34.858+0000 7fac12a52640 -1 reset not still connected to 0x5574d5072c30 2024-09-12T06:45:35.313 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:35.310+0000 7fac12a52640 -1 reset not still connected to 0x55749b1cb5f0 2024-09-12T06:45:35.313 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:35.312+0000 7fac12a52640 -1 reset not still connected to 0x55749b1cb860 2024-09-12T06:45:35.317 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:35.318+0000 7fac12a52640 -1 reset not still connected to 0x55749b1cbba0 2024-09-12T06:45:35.317 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:35.318+0000 7fac12a52640 -1 reset not still connected to 0x55749b2bf110 2024-09-12T06:45:35.317 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:35.318+0000 7fac12a52640 -1 reset not still connected to 0x55749b2bfee0 2024-09-12T06:45:35.318 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:35.318+0000 7fac12a52640 -1 reset not still connected to 0x55749c9e9380 2024-09-12T06:45:35.318 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:35.318+0000 7fac12a52640 -1 reset not still connected to 0x55749ca4d040 2024-09-12T06:45:35.318 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:35.318+0000 7fac12a52640 -1 reset not still connected to 0x55749d3c4dd0 2024-09-12T06:45:35.318 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:35.318+0000 7fac12a52640 -1 reset not still connected to 0x55749d402410 2024-09-12T06:45:35.318 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:35.318+0000 7fac12a52640 -1 reset not still connected to 0x55749d402680 2024-09-12T06:45:35.318 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:35.318+0000 7fac12a52640 -1 reset not still connected to 0x55749d403ba0 2024-09-12T06:45:35.318 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:35.318+0000 7fac12a52640 -1 reset not still connected to 0x55749d7ca270 2024-09-12T06:45:35.318 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:35.319+0000 7fac12a52640 -1 reset not still connected to 0x55749d7ca8f0 2024-09-12T06:45:35.318 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:35.319+0000 7fac12a52640 -1 reset not still connected to 0x55749e05cf70 2024-09-12T06:45:35.322 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:35.323+0000 7fac12a52640 -1 reset not still connected to 0x5574a2e61d40 2024-09-12T06:45:35.322 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:35.323+0000 7fac12a52640 -1 reset not still connected to 0x5574a2fa7a00 2024-09-12T06:45:35.322 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:35.323+0000 7fac12a52640 -1 reset not still connected to 0x5574a62b6d00 2024-09-12T06:45:35.322 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:35.323+0000 7fac12a52640 -1 reset not still connected to 0x5574bdabf110 2024-09-12T06:45:35.323 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:35.323+0000 7fac12a52640 -1 reset not still connected to 0x5574bdabf1e0 2024-09-12T06:45:35.323 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:35.323+0000 7fac12a52640 -1 reset not still connected to 0x5574bdabfd40 2024-09-12T06:45:35.323 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:35.323+0000 7fac12a52640 -1 reset not still connected to 0x5574cf6f9ba0 2024-09-12T06:45:35.323 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:35.323+0000 7fac12a52640 -1 reset not still connected to 0x5574cfee1c70 2024-09-12T06:45:35.323 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:35.323+0000 7fac12a52640 -1 reset not still connected to 0x5574d1d6da00 2024-09-12T06:45:35.323 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:35.323+0000 7fac12a52640 -1 reset not still connected to 0x5574d5072c30 2024-09-12T06:45:35.522 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:100 IOs, 43777024 bytes 2024-09-12T06:45:36.502 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:45:36.504+0000 7f80d15bb640 -1 reset not still connected to 0x556b05b5e680 2024-09-12T06:45:36.503 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:45:36.504+0000 7f80d15bb640 -1 reset not still connected to 0x556b07468000 2024-09-12T06:45:36.503 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:45:36.504+0000 7f80d15bb640 -1 reset not still connected to 0x556b0bcea9c0 2024-09-12T06:45:36.503 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:45:36.504+0000 7f80d15bb640 -1 reset not still connected to 0x556b33ff76c0 2024-09-12T06:45:38.102 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ error_code=124 2024-09-12T06:45:38.102 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set -e 2024-09-12T06:45:38.102 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T06:45:38.103 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:45:38.103 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_pool_images cluster2 mirror 50 2024-09-12T06:45:38.103 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:45:38.103 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:45:38.103 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_count=50 2024-09-12T06:45:38.103 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:45:38.103 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local count 2024-09-12T06:45:38.103 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local last_count=0 2024-09-12T06:45:38.103 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:45:38.103 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 1 40 2024-09-12T06:45:38.104 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in `seq 1 40` 2024-09-12T06:45:38.104 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 1 -ne 1 2024-09-12T06:45:38.104 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep 'images: ' 2024-09-12T06:45:38.104 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2024-09-12T06:45:38.105 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut '-d ' -f 2 2024-09-12T06:45:38.155 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: failed to query services: (13) Permission denied 2024-09-12T06:45:38.159 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ count=50 2024-09-12T06:45:38.159 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 50 = 50 2024-09-12T06:45:38.159 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:45:38.159 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_pool_healthy cluster2 mirror 2024-09-12T06:45:38.159 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:45:38.159 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:45:38.160 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:45:38.160 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state 2024-09-12T06:45:38.160 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 1 40 2024-09-12T06:45:38.160 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in `seq 1 40` 2024-09-12T06:45:38.160 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 1 -ne 1 2024-09-12T06:45:38.161 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2024-09-12T06:45:38.161 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep 'image health:' 2024-09-12T06:45:38.161 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut '-d ' -f 3 2024-09-12T06:45:38.208 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: failed to query services: (13) Permission denied 2024-09-12T06:45:38.212 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ state=WARNING 2024-09-12T06:45:38.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test WARNING = ERROR 2024-09-12T06:45:38.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test WARNING = OK 2024-09-12T06:45:38.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in `seq 1 40` 2024-09-12T06:45:38.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 2 -ne 1 2024-09-12T06:45:38.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 30 2024-09-12T06:45:38.696 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:38.693+0000 7fa325ae1640 -1 reset not still connected to 0x55f85a4fc9c0 2024-09-12T06:45:38.696 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:38.693+0000 7fa325ae1640 -1 reset not still connected to 0x55f85a4fcd00 2024-09-12T06:45:38.696 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:38.697+0000 7fa325ae1640 -1 reset not still connected to 0x55f85a4fcdd0 2024-09-12T06:45:38.696 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:38.697+0000 7fa325ae1640 -1 reset not still connected to 0x55f85bac3520 2024-09-12T06:45:38.696 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:38.697+0000 7fa325ae1640 -1 reset not still connected to 0x55f85cc67040 2024-09-12T06:45:38.696 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:38.697+0000 7fa325ae1640 -1 reset not still connected to 0x55f863611110 2024-09-12T06:45:38.697 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:38.697+0000 7fa325ae1640 -1 reset not still connected to 0x55f869fdc0d0 2024-09-12T06:45:38.697 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:38.697+0000 7fa325ae1640 -1 reset not still connected to 0x55f869fdc340 2024-09-12T06:45:38.702 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:38.699+0000 7fa325ae1640 -1 reset not still connected to 0x55f86ee2fe10 2024-09-12T06:45:38.702 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:38.699+0000 7fa325ae1640 -1 reset not still connected to 0x55f878a54410 2024-09-12T06:45:38.702 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:38.699+0000 7fa325ae1640 -1 reset not still connected to 0x55f88041e270 2024-09-12T06:45:38.702 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:38.700+0000 7fa325ae1640 -1 reset not still connected to 0x55f880b4e4e0 2024-09-12T06:45:38.702 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:38.700+0000 7fa325ae1640 -1 reset not still connected to 0x55f88417c410 2024-09-12T06:45:38.702 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:38.700+0000 7fa325ae1640 -1 reset not still connected to 0x55f884309c70 2024-09-12T06:45:38.702 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:38.700+0000 7fa325ae1640 -1 reset not still connected to 0x55f8849a1040 2024-09-12T06:45:38.702 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:38.700+0000 7fa325ae1640 -1 reset not still connected to 0x55f8849a11e0 2024-09-12T06:45:38.702 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:38.700+0000 7fa325ae1640 -1 reset not still connected to 0x55f8863b4340 2024-09-12T06:45:38.703 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:38.701+0000 7fa325ae1640 -1 reset not still connected to 0x55f8863b4680 2024-09-12T06:45:38.703 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:38.705+0000 7fa325ae1640 -1 reset not still connected to 0x55f8863b5790 2024-09-12T06:45:38.703 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:38.705+0000 7fa325ae1640 -1 reset not still connected to 0x55f88b2cd5f0 2024-09-12T06:45:38.703 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:38.705+0000 7fa325ae1640 -1 reset not still connected to 0x55f890e69a00 2024-09-12T06:45:38.703 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:38.705+0000 7fa325ae1640 -1 reset not still connected to 0x55f8926ce0d0 2024-09-12T06:45:38.703 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:38.705+0000 7fa325ae1640 -1 reset not still connected to 0x55f8926ce1a0 2024-09-12T06:45:38.703 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:38.705+0000 7fa325ae1640 -1 reset not still connected to 0x55f896243040 2024-09-12T06:45:38.783 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:38.782+0000 7fac12a52640 -1 reset not still connected to 0x55749b1ca5b0 2024-09-12T06:45:38.784 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:38.783+0000 7fac12a52640 -1 reset not still connected to 0x55749b1cbe10 2024-09-12T06:45:38.784 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:38.783+0000 7fac12a52640 -1 reset not still connected to 0x55749b1cbee0 2024-09-12T06:45:38.784 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:38.783+0000 7fac12a52640 -1 reset not still connected to 0x55749b2be0d0 2024-09-12T06:45:38.784 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:38.783+0000 7fac12a52640 -1 reset not still connected to 0x55749b2be8f0 2024-09-12T06:45:38.784 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:38.783+0000 7fac12a52640 -1 reset not still connected to 0x55749b2bed00 2024-09-12T06:45:38.784 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:38.783+0000 7fac12a52640 -1 reset not still connected to 0x55749b2bedd0 2024-09-12T06:45:38.784 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:38.783+0000 7fac12a52640 -1 reset not still connected to 0x55749b2bf6c0 2024-09-12T06:45:38.784 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:38.783+0000 7fac12a52640 -1 reset not still connected to 0x55749b2bfc70 2024-09-12T06:45:38.784 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:38.783+0000 7fac12a52640 -1 reset not still connected to 0x55749c9a7860 2024-09-12T06:45:38.784 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:38.783+0000 7fac12a52640 -1 reset not still connected to 0x55749ca4d790 2024-09-12T06:45:38.785 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:38.784+0000 7fac12a52640 -1 reset not still connected to 0x55749d4025b0 2024-09-12T06:45:38.785 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:38.784+0000 7fac12a52640 -1 reset not still connected to 0x55749e3c4750 2024-09-12T06:45:38.785 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:38.784+0000 7fac12a52640 -1 reset not still connected to 0x55749e3c48f0 2024-09-12T06:45:38.785 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:38.784+0000 7fac12a52640 -1 reset not still connected to 0x55749e3c4d00 2024-09-12T06:45:38.785 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:38.784+0000 7fac12a52640 -1 reset not still connected to 0x5574b3849380 2024-09-12T06:45:40.376 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:40.369+0000 7fac12a52640 -1 reset not still connected to 0x5574a2fa7a00 2024-09-12T06:45:40.376 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:40.369+0000 7fac12a52640 -1 reset not still connected to 0x5574a62b6d00 2024-09-12T06:45:40.376 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:40.371+0000 7fac12a52640 -1 reset not still connected to 0x5574bdabf1e0 2024-09-12T06:45:40.376 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:40.371+0000 7fac12a52640 -1 reset not still connected to 0x5574cf6f9ba0 2024-09-12T06:45:40.379 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:40.380+0000 7fac12a52640 -1 reset not still connected to 0x5574d5072c30 2024-09-12T06:45:40.406 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:40.407+0000 7fac12a52640 -1 reset not still connected to 0x55749b2bfee0 2024-09-12T06:45:40.407 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:40.407+0000 7fac12a52640 -1 reset not still connected to 0x55749c9e9380 2024-09-12T06:45:40.407 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:40.407+0000 7fac12a52640 -1 reset not still connected to 0x55749ca4d040 2024-09-12T06:45:40.407 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:40.407+0000 7fac12a52640 -1 reset not still connected to 0x55749d3c4dd0 2024-09-12T06:45:40.407 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:40.407+0000 7fac12a52640 -1 reset not still connected to 0x55749d402410 2024-09-12T06:45:40.407 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:40.407+0000 7fac12a52640 -1 reset not still connected to 0x55749d402680 2024-09-12T06:45:40.407 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:40.407+0000 7fac12a52640 -1 reset not still connected to 0x55749d403ba0 2024-09-12T06:45:40.407 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:40.407+0000 7fac12a52640 -1 reset not still connected to 0x55749d7ca270 2024-09-12T06:45:40.410 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:40.411+0000 7fac12a52640 -1 reset not still connected to 0x55749e05cf70 2024-09-12T06:45:40.414 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:40.415+0000 7fac12a52640 -1 reset not still connected to 0x5574a2e61d40 2024-09-12T06:45:40.414 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:40.415+0000 7fac12a52640 -1 reset not still connected to 0x5574a2fa7a00 2024-09-12T06:45:40.414 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:40.415+0000 7fac12a52640 -1 reset not still connected to 0x5574a62b6d00 2024-09-12T06:45:40.414 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:40.415+0000 7fac12a52640 -1 reset not still connected to 0x5574bdabf110 2024-09-12T06:45:40.414 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:40.415+0000 7fac12a52640 -1 reset not still connected to 0x5574bdabf1e0 2024-09-12T06:45:40.414 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:40.415+0000 7fac12a52640 -1 reset not still connected to 0x5574bdabfd40 2024-09-12T06:45:40.414 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:40.415+0000 7fac12a52640 -1 reset not still connected to 0x5574cf6f9ba0 2024-09-12T06:45:40.414 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:40.415+0000 7fac12a52640 -1 reset not still connected to 0x5574cfee1c70 2024-09-12T06:45:40.414 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:40.415+0000 7fac12a52640 -1 reset not still connected to 0x5574d5072c30 2024-09-12T06:45:40.756 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:40.756+0000 7fac12a52640 -1 reset not still connected to 0x55749ca4d040 2024-09-12T06:45:40.756 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:40.756+0000 7fac12a52640 -1 reset not still connected to 0x55749d3c4dd0 2024-09-12T06:45:40.756 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:40.756+0000 7fac12a52640 -1 reset not still connected to 0x55749d402410 2024-09-12T06:45:40.756 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:40.756+0000 7fac12a52640 -1 reset not still connected to 0x55749d402680 2024-09-12T06:45:40.756 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:40.756+0000 7fac12a52640 -1 reset not still connected to 0x55749d403ba0 2024-09-12T06:45:40.756 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:40.756+0000 7fac12a52640 -1 reset not still connected to 0x55749d7ca270 2024-09-12T06:45:40.758 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:40.759+0000 7fac12a52640 -1 reset not still connected to 0x5574a2e61d40 2024-09-12T06:45:40.758 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:40.759+0000 7fac12a52640 -1 reset not still connected to 0x5574a2fa7a00 2024-09-12T06:45:40.758 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:40.759+0000 7fac12a52640 -1 reset not still connected to 0x5574a62b6d00 2024-09-12T06:45:40.758 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:40.759+0000 7fac12a52640 -1 reset not still connected to 0x5574bdabf110 2024-09-12T06:45:40.758 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:40.759+0000 7fac12a52640 -1 reset not still connected to 0x5574bdabf1e0 2024-09-12T06:45:40.758 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:40.759+0000 7fac12a52640 -1 reset not still connected to 0x5574cf6f9ba0 2024-09-12T06:45:40.758 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:40.759+0000 7fac12a52640 -1 reset not still connected to 0x5574cfee1c70 2024-09-12T06:45:40.758 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:40.759+0000 7fac12a52640 -1 reset not still connected to 0x5574d5072c30 2024-09-12T06:45:42.041 INFO:tasks.ceph.cluster2.osd.0.smithi152.stderr:2024-09-12T06:45:42.042+0000 7fd61689f640 -1 reset not still connected to 0x55e7772cba00 2024-09-12T06:45:42.042 INFO:tasks.ceph.cluster2.osd.0.smithi152.stderr:2024-09-12T06:45:42.043+0000 7fd61689f640 -1 reset not still connected to 0x55e77b1fcc30 2024-09-12T06:45:42.042 INFO:tasks.ceph.cluster2.osd.0.smithi152.stderr:2024-09-12T06:45:42.043+0000 7fd61689f640 -1 reset not still connected to 0x55e77b1fcd00 2024-09-12T06:45:42.042 INFO:tasks.ceph.cluster2.osd.0.smithi152.stderr:2024-09-12T06:45:42.043+0000 7fd61689f640 -1 reset not still connected to 0x55e77c768680 2024-09-12T06:45:42.047 INFO:tasks.ceph.cluster2.osd.0.smithi152.stderr:2024-09-12T06:45:42.048+0000 7fd61689f640 -1 reset not still connected to 0x55e77d6f2d00 2024-09-12T06:45:42.047 INFO:tasks.ceph.cluster2.osd.0.smithi152.stderr:2024-09-12T06:45:42.048+0000 7fd61689f640 -1 reset not still connected to 0x55e77d6f3a00 2024-09-12T06:45:42.047 INFO:tasks.ceph.cluster2.osd.0.smithi152.stderr:2024-09-12T06:45:42.048+0000 7fd61689f640 -1 reset not still connected to 0x55e782f70750 2024-09-12T06:45:42.047 INFO:tasks.ceph.cluster2.osd.0.smithi152.stderr:2024-09-12T06:45:42.048+0000 7fd61689f640 -1 reset not still connected to 0x55e7852149c0 2024-09-12T06:45:42.047 INFO:tasks.ceph.cluster2.osd.0.smithi152.stderr:2024-09-12T06:45:42.048+0000 7fd61689f640 -1 reset not still connected to 0x55e7852155f0 2024-09-12T06:45:42.110 INFO:tasks.ceph.cluster2.osd.1.smithi152.stderr:2024-09-12T06:45:42.112+0000 7f33ea860640 -1 bdev(0x55c73b90ec00 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 4 2024-09-12T06:45:42.110 INFO:tasks.ceph.cluster2.osd.1.smithi152.stderr:2024-09-12T06:45:42.112+0000 7f33e885c640 -1 bdev(0x55c73b90ec00 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 4 2024-09-12T06:45:43.629 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:45:43.630+0000 7f80d15bb640 -1 reset not still connected to 0x556b043adba0 2024-09-12T06:45:43.629 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:45:43.630+0000 7f80d15bb640 -1 reset not still connected to 0x556b044000d0 2024-09-12T06:45:43.632 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:45:43.632+0000 7f80d15bb640 -1 reset not still connected to 0x556b0c2c6750 2024-09-12T06:45:43.632 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:45:43.634+0000 7f80d15bb640 -1 reset not still connected to 0x556b14db6a90 2024-09-12T06:45:43.632 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:45:43.634+0000 7f80d15bb640 -1 reset not still connected to 0x556b32cfef70 2024-09-12T06:45:43.632 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:45:43.634+0000 7f80d15bb640 -1 reset not still connected to 0x556b3e8a9790 2024-09-12T06:45:44.811 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:44.812+0000 7fa325ae1640 -1 reset not still connected to 0x55f8926ce1a0 2024-09-12T06:45:44.812 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:44.812+0000 7fa325ae1640 -1 reset not still connected to 0x55f896243040 2024-09-12T06:45:44.826 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:44.827+0000 7fac12a52640 -1 reset not still connected to 0x55749aea7790 2024-09-12T06:45:44.826 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:44.827+0000 7fac12a52640 -1 reset not still connected to 0x55749b1cbc70 2024-09-12T06:45:46.672 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:46.673+0000 7fac12a52640 -1 reset not still connected to 0x55749d403ba0 2024-09-12T06:45:46.673 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:46.674+0000 7fac12a52640 -1 reset not still connected to 0x55749d7ca270 2024-09-12T06:45:46.677 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:46.676+0000 7fac12a52640 -1 reset not still connected to 0x55749d7ca8f0 2024-09-12T06:45:46.677 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:46.676+0000 7fac12a52640 -1 reset not still connected to 0x55749e05cf70 2024-09-12T06:45:46.677 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:46.677+0000 7fac12a52640 -1 reset not still connected to 0x5574a2e61d40 2024-09-12T06:45:46.677 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:46.677+0000 7fac12a52640 -1 reset not still connected to 0x5574a2fa7a00 2024-09-12T06:45:46.677 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:46.677+0000 7fac12a52640 -1 reset not still connected to 0x5574a62b6d00 2024-09-12T06:45:46.677 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:46.677+0000 7fac12a52640 -1 reset not still connected to 0x5574b3848d00 2024-09-12T06:45:46.677 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:46.677+0000 7fac12a52640 -1 reset not still connected to 0x5574bdabf110 2024-09-12T06:45:46.677 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:46.677+0000 7fac12a52640 -1 reset not still connected to 0x5574bdabf1e0 2024-09-12T06:45:46.677 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:46.677+0000 7fac12a52640 -1 reset not still connected to 0x5574bdabfd40 2024-09-12T06:45:46.677 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:46.677+0000 7fac12a52640 -1 reset not still connected to 0x5574cf6f9ba0 2024-09-12T06:45:46.677 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:46.677+0000 7fac12a52640 -1 reset not still connected to 0x5574cfee1c70 2024-09-12T06:45:46.677 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:46.677+0000 7fac12a52640 -1 reset not still connected to 0x5574d1d6da00 2024-09-12T06:45:46.677 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:45:46.677+0000 7fac12a52640 -1 reset not still connected to 0x5574d5072c30 2024-09-12T06:45:49.079 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:45:49.080+0000 7f80d15bb640 -1 reset not still connected to 0x556b04ee6b60 2024-09-12T06:45:49.079 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:45:49.080+0000 7f80d15bb640 -1 reset not still connected to 0x556b05b5e680 2024-09-12T06:45:49.080 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:45:49.080+0000 7f80d15bb640 -1 reset not still connected to 0x556b07468000 2024-09-12T06:45:49.080 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:45:49.080+0000 7f80d15bb640 -1 reset not still connected to 0x556b0bcea9c0 2024-09-12T06:45:49.080 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:45:49.080+0000 7f80d15bb640 -1 reset not still connected to 0x556b33ff76c0 2024-09-12T06:45:49.817 INFO:tasks.ceph.cluster2.osd.2.smithi152.stderr:2024-09-12T06:45:49.819+0000 7f4789beb640 -1 reset not still connected to 0x56163d519e10 2024-09-12T06:45:49.818 INFO:tasks.ceph.cluster2.osd.2.smithi152.stderr:2024-09-12T06:45:49.819+0000 7f4789beb640 -1 reset not still connected to 0x56163df360d0 2024-09-12T06:45:49.818 INFO:tasks.ceph.cluster2.osd.2.smithi152.stderr:2024-09-12T06:45:49.819+0000 7f4789beb640 -1 reset not still connected to 0x56163df36270 2024-09-12T06:45:51.128 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:51.129+0000 7fa325ae1640 -1 reset not still connected to 0x55f85d61d380 2024-09-12T06:45:51.128 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:51.129+0000 7fa325ae1640 -1 reset not still connected to 0x55f8636984e0 2024-09-12T06:45:51.128 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:51.129+0000 7fa325ae1640 -1 reset not still connected to 0x55f863699040 2024-09-12T06:45:51.128 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:51.129+0000 7fa325ae1640 -1 reset not still connected to 0x55f86ee2e340 2024-09-12T06:45:51.128 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:51.129+0000 7fa325ae1640 -1 reset not still connected to 0x55f87faff040 2024-09-12T06:45:51.129 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:51.129+0000 7fa325ae1640 -1 reset not still connected to 0x55f88c6bfa00 2024-09-12T06:45:51.129 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:51.129+0000 7fa325ae1640 -1 reset not still connected to 0x55f8926ce750 2024-09-12T06:45:51.129 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:51.129+0000 7fa325ae1640 -1 reset not still connected to 0x55f8927168f0 2024-09-12T06:45:51.129 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:51.129+0000 7fa325ae1640 -1 reset not still connected to 0x55f892fe4b60 2024-09-12T06:45:53.115 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:45:53.116+0000 7f80d15bb640 -1 reset not still connected to 0x556b07468000 2024-09-12T06:45:53.115 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:45:53.116+0000 7f80d15bb640 -1 reset not still connected to 0x556b0bcea9c0 2024-09-12T06:45:53.116 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:45:53.118+0000 7f80d15bb640 -1 reset not still connected to 0x556b33ff76c0 2024-09-12T06:45:56.979 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:56.980+0000 7fa325ae1640 -1 reset not still connected to 0x55f8849a1040 2024-09-12T06:45:56.979 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:56.980+0000 7fa325ae1640 -1 reset not still connected to 0x55f8849a11e0 2024-09-12T06:45:56.984 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:56.986+0000 7fa325ae1640 -1 reset not still connected to 0x55f8863b5790 2024-09-12T06:45:56.984 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:56.986+0000 7fa325ae1640 -1 reset not still connected to 0x55f88b2cd5f0 2024-09-12T06:45:56.984 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:56.986+0000 7fa325ae1640 -1 reset not still connected to 0x55f890e69a00 2024-09-12T06:45:56.989 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:56.990+0000 7fa325ae1640 -1 reset not still connected to 0x55f8926ce0d0 2024-09-12T06:45:56.989 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:56.990+0000 7fa325ae1640 -1 reset not still connected to 0x55f8926ce1a0 2024-09-12T06:45:56.989 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:56.990+0000 7fa325ae1640 -1 reset not still connected to 0x55f896243040 2024-09-12T06:45:57.397 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:57.398+0000 7fa325ae1640 -1 reset not still connected to 0x55f8849a1040 2024-09-12T06:45:57.398 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:57.398+0000 7fa325ae1640 -1 reset not still connected to 0x55f8849a11e0 2024-09-12T06:45:57.407 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:57.408+0000 7fa325ae1640 -1 reset not still connected to 0x55f8863b4680 2024-09-12T06:45:57.408 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:57.408+0000 7fa325ae1640 -1 reset not still connected to 0x55f8863b5790 2024-09-12T06:45:57.408 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:57.408+0000 7fa325ae1640 -1 reset not still connected to 0x55f88b2cd5f0 2024-09-12T06:45:57.408 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:57.408+0000 7fa325ae1640 -1 reset not still connected to 0x55f890e69a00 2024-09-12T06:45:57.408 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:57.408+0000 7fa325ae1640 -1 reset not still connected to 0x55f8926ce0d0 2024-09-12T06:45:57.408 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:57.408+0000 7fa325ae1640 -1 reset not still connected to 0x55f8926ce1a0 2024-09-12T06:45:57.408 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:45:57.408+0000 7fa325ae1640 -1 reset not still connected to 0x55f896243040 2024-09-12T06:46:02.322 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:02.324+0000 7fa325ae1640 -1 reset not still connected to 0x55f85a4fcdd0 2024-09-12T06:46:02.322 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:02.324+0000 7fa325ae1640 -1 reset not still connected to 0x55f85bac3520 2024-09-12T06:46:02.329 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:02.329+0000 7fa325ae1640 -1 reset not still connected to 0x55f85cc67040 2024-09-12T06:46:02.329 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:02.329+0000 7fa325ae1640 -1 reset not still connected to 0x55f863611110 2024-09-12T06:46:02.329 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:02.329+0000 7fa325ae1640 -1 reset not still connected to 0x55f869fdc0d0 2024-09-12T06:46:02.330 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:02.331+0000 7fa325ae1640 -1 reset not still connected to 0x55f869fdc340 2024-09-12T06:46:02.330 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:02.331+0000 7fa325ae1640 -1 reset not still connected to 0x55f86ee2fe10 2024-09-12T06:46:02.330 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:02.331+0000 7fa325ae1640 -1 reset not still connected to 0x55f878a54410 2024-09-12T06:46:02.330 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:02.331+0000 7fa325ae1640 -1 reset not still connected to 0x55f88041e270 2024-09-12T06:46:02.330 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:02.331+0000 7fa325ae1640 -1 reset not still connected to 0x55f880b4e4e0 2024-09-12T06:46:02.330 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:02.331+0000 7fa325ae1640 -1 reset not still connected to 0x55f88417c410 2024-09-12T06:46:02.334 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:02.335+0000 7fa325ae1640 -1 reset not still connected to 0x55f884309c70 2024-09-12T06:46:02.334 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:02.335+0000 7fa325ae1640 -1 reset not still connected to 0x55f8849a1040 2024-09-12T06:46:02.334 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:02.335+0000 7fa325ae1640 -1 reset not still connected to 0x55f8849a11e0 2024-09-12T06:46:02.334 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:02.335+0000 7fa325ae1640 -1 reset not still connected to 0x55f8863b4340 2024-09-12T06:46:02.334 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:02.335+0000 7fa325ae1640 -1 reset not still connected to 0x55f8863b4680 2024-09-12T06:46:02.334 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:02.335+0000 7fa325ae1640 -1 reset not still connected to 0x55f8863b5790 2024-09-12T06:46:02.334 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:02.335+0000 7fa325ae1640 -1 reset not still connected to 0x55f88b2cd5f0 2024-09-12T06:46:02.334 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:02.335+0000 7fa325ae1640 -1 reset not still connected to 0x55f890e69a00 2024-09-12T06:46:02.335 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:02.337+0000 7fa325ae1640 -1 reset not still connected to 0x55f8926ce0d0 2024-09-12T06:46:02.335 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:02.337+0000 7fa325ae1640 -1 reset not still connected to 0x55f8926ce1a0 2024-09-12T06:46:02.336 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:02.337+0000 7fa325ae1640 -1 reset not still connected to 0x55f896243040 2024-09-12T06:46:02.375 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:02.375+0000 7fa325ae1640 -1 reset not still connected to 0x55f859179380 2024-09-12T06:46:02.382 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:02.382+0000 7fa325ae1640 -1 reset not still connected to 0x55f85a304a90 2024-09-12T06:46:02.382 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:02.382+0000 7fa325ae1640 -1 reset not still connected to 0x55f85ac224e0 2024-09-12T06:46:02.382 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:02.382+0000 7fa325ae1640 -1 reset not still connected to 0x55f85ac229c0 2024-09-12T06:46:02.382 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:02.382+0000 7fa325ae1640 -1 reset not still connected to 0x55f85ba925b0 2024-09-12T06:46:02.385 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:02.386+0000 7fa325ae1640 -1 reset not still connected to 0x55f85ba92d00 2024-09-12T06:46:02.385 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:02.386+0000 7fa325ae1640 -1 reset not still connected to 0x55f85bac29c0 2024-09-12T06:46:02.387 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:02.386+0000 7fa325ae1640 -1 reset not still connected to 0x55f85bac2b60 2024-09-12T06:46:02.389 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:02.388+0000 7fa325ae1640 -1 reset not still connected to 0x55f85cc67860 2024-09-12T06:46:02.389 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:02.388+0000 7fa325ae1640 -1 reset not still connected to 0x55f85cc67930 2024-09-12T06:46:02.389 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:02.389+0000 7fa325ae1640 -1 reset not still connected to 0x55f85d61d380 2024-09-12T06:46:02.389 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:02.389+0000 7fa325ae1640 -1 reset not still connected to 0x55f8636984e0 2024-09-12T06:46:02.389 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:02.389+0000 7fa325ae1640 -1 reset not still connected to 0x55f863699040 2024-09-12T06:46:02.389 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:02.389+0000 7fa325ae1640 -1 reset not still connected to 0x55f86ee2e340 2024-09-12T06:46:02.389 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:02.389+0000 7fa325ae1640 -1 reset not still connected to 0x55f87faff040 2024-09-12T06:46:02.389 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:02.389+0000 7fa325ae1640 -1 reset not still connected to 0x55f88c6bfa00 2024-09-12T06:46:02.389 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:02.389+0000 7fa325ae1640 -1 reset not still connected to 0x55f8926ce750 2024-09-12T06:46:02.390 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:02.389+0000 7fa325ae1640 -1 reset not still connected to 0x55f8927168f0 2024-09-12T06:46:02.390 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:02.389+0000 7fa325ae1640 -1 reset not still connected to 0x55f892fe4b60 2024-09-12T06:46:03.554 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:03.544+0000 7fa325ae1640 -1 reset not still connected to 0x55f85a304a90 2024-09-12T06:46:03.554 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:03.546+0000 7fa325ae1640 -1 reset not still connected to 0x55f85ac224e0 2024-09-12T06:46:03.554 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:03.551+0000 7fa325ae1640 -1 reset not still connected to 0x55f85ba92340 2024-09-12T06:46:03.554 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:03.551+0000 7fa325ae1640 -1 reset not still connected to 0x55f85bac29c0 2024-09-12T06:46:03.554 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:03.551+0000 7fa325ae1640 -1 reset not still connected to 0x55f85bac2b60 2024-09-12T06:46:03.554 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:03.551+0000 7fa325ae1640 -1 reset not still connected to 0x55f85cc67860 2024-09-12T06:46:03.554 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:03.551+0000 7fa325ae1640 -1 reset not still connected to 0x55f85cc67930 2024-09-12T06:46:03.554 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:03.551+0000 7fa325ae1640 -1 reset not still connected to 0x55f85d61d380 2024-09-12T06:46:03.554 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:03.551+0000 7fa325ae1640 -1 reset not still connected to 0x55f86ee2e340 2024-09-12T06:46:03.555 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:03.551+0000 7fa325ae1640 -1 reset not still connected to 0x55f87faff040 2024-09-12T06:46:03.555 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:03.551+0000 7fa325ae1640 -1 reset not still connected to 0x55f88c6bfa00 2024-09-12T06:46:03.555 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:03.553+0000 7fa325ae1640 -1 reset not still connected to 0x55f8926ce750 2024-09-12T06:46:03.555 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:03.553+0000 7fa325ae1640 -1 reset not still connected to 0x55f8927168f0 2024-09-12T06:46:05.455 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:05.456+0000 7fac12a52640 -1 reset not still connected to 0x5574a2fa7a00 2024-09-12T06:46:05.455 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:05.456+0000 7fac12a52640 -1 reset not still connected to 0x5574a62b6d00 2024-09-12T06:46:05.457 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:05.458+0000 7fac12a52640 -1 reset not still connected to 0x5574bdabf1e0 2024-09-12T06:46:05.457 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:05.458+0000 7fac12a52640 -1 reset not still connected to 0x5574cf6f9ba0 2024-09-12T06:46:05.459 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:05.461+0000 7fac12a52640 -1 reset not still connected to 0x5574d5072c30 2024-09-12T06:46:08.296 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2024-09-12T06:46:08.296 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep 'image health:' 2024-09-12T06:46:08.297 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut '-d ' -f 3 2024-09-12T06:46:08.344 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: failed to query services: (13) Permission denied 2024-09-12T06:46:08.348 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ state=WARNING 2024-09-12T06:46:08.348 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test WARNING = ERROR 2024-09-12T06:46:08.348 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test WARNING = OK 2024-09-12T06:46:08.348 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in `seq 1 40` 2024-09-12T06:46:08.348 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 3 -ne 1 2024-09-12T06:46:08.348 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 30 2024-09-12T06:46:09.466 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:46:09.466+0000 7f80d15bb640 -1 reset not still connected to 0x556b14db6a90 2024-09-12T06:46:09.466 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:46:09.466+0000 7f80d15bb640 -1 reset not still connected to 0x556b32cfef70 2024-09-12T06:46:09.466 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:46:09.467+0000 7f80d15bb640 -1 reset not still connected to 0x556b3e8a9790 2024-09-12T06:46:09.499 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:46:09.500+0000 7f80d15bb640 -1 reset not still connected to 0x556b14db6a90 2024-09-12T06:46:09.499 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:46:09.500+0000 7f80d15bb640 -1 reset not still connected to 0x556b32cfef70 2024-09-12T06:46:09.499 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:46:09.500+0000 7f80d15bb640 -1 reset not still connected to 0x556b3e8a9790 2024-09-12T06:46:12.760 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:12.760+0000 7fa325ae1640 -1 reset not still connected to 0x55f8863b5790 2024-09-12T06:46:12.761 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:12.760+0000 7fa325ae1640 -1 reset not still connected to 0x55f88b2cd5f0 2024-09-12T06:46:12.761 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:12.760+0000 7fa325ae1640 -1 reset not still connected to 0x55f8926ce0d0 2024-09-12T06:46:12.761 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:12.760+0000 7fa325ae1640 -1 reset not still connected to 0x55f8926ce1a0 2024-09-12T06:46:12.761 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:12.762+0000 7fa325ae1640 -1 reset not still connected to 0x55f896243040 2024-09-12T06:46:14.069 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:14.069+0000 7fa325ae1640 -1 reset not still connected to 0x55f8863b4340 2024-09-12T06:46:14.069 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:14.069+0000 7fa325ae1640 -1 reset not still connected to 0x55f8863b4680 2024-09-12T06:46:14.069 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:14.069+0000 7fa325ae1640 -1 reset not still connected to 0x55f8863b5790 2024-09-12T06:46:14.069 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:14.069+0000 7fa325ae1640 -1 reset not still connected to 0x55f88b2cd5f0 2024-09-12T06:46:14.070 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:14.069+0000 7fa325ae1640 -1 reset not still connected to 0x55f890e69a00 2024-09-12T06:46:14.070 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:14.070+0000 7fa325ae1640 -1 reset not still connected to 0x55f8926ce0d0 2024-09-12T06:46:14.070 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:14.070+0000 7fa325ae1640 -1 reset not still connected to 0x55f8926ce1a0 2024-09-12T06:46:14.070 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:14.070+0000 7fa325ae1640 -1 reset not still connected to 0x55f896243040 2024-09-12T06:46:16.858 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:16.853+0000 7fac12a52640 -1 reset not still connected to 0x55749d402410 2024-09-12T06:46:16.858 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:16.853+0000 7fac12a52640 -1 reset not still connected to 0x55749d402680 2024-09-12T06:46:16.858 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:16.853+0000 7fac12a52640 -1 reset not still connected to 0x55749d403ba0 2024-09-12T06:46:16.858 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:16.853+0000 7fac12a52640 -1 reset not still connected to 0x55749d7ca270 2024-09-12T06:46:16.858 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:16.854+0000 7fac12a52640 -1 reset not still connected to 0x55749e05cf70 2024-09-12T06:46:16.858 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:16.855+0000 7fac12a52640 -1 reset not still connected to 0x5574a2e61d40 2024-09-12T06:46:16.858 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:16.858+0000 7fac12a52640 -1 reset not still connected to 0x5574a2fa7a00 2024-09-12T06:46:16.858 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:16.858+0000 7fac12a52640 -1 reset not still connected to 0x5574a62b6d00 2024-09-12T06:46:16.859 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:16.858+0000 7fac12a52640 -1 reset not still connected to 0x5574bdabf110 2024-09-12T06:46:16.859 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:16.859+0000 7fac12a52640 -1 reset not still connected to 0x5574bdabf1e0 2024-09-12T06:46:16.859 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:16.860+0000 7fac12a52640 -1 reset not still connected to 0x5574bdabfd40 2024-09-12T06:46:16.859 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:16.860+0000 7fac12a52640 -1 reset not still connected to 0x5574cf6f9ba0 2024-09-12T06:46:16.859 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:16.860+0000 7fac12a52640 -1 reset not still connected to 0x5574cfee1c70 2024-09-12T06:46:16.862 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:16.864+0000 7fac12a52640 -1 reset not still connected to 0x5574d1d6da00 2024-09-12T06:46:16.862 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:16.864+0000 7fac12a52640 -1 reset not still connected to 0x5574d5072c30 2024-09-12T06:46:19.259 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:19.252+0000 7fac12a52640 -1 reset not still connected to 0x55749b2bf110 2024-09-12T06:46:19.260 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:19.252+0000 7fac12a52640 -1 reset not still connected to 0x55749b2bfee0 2024-09-12T06:46:19.261 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:19.256+0000 7fac12a52640 -1 reset not still connected to 0x55749d402410 2024-09-12T06:46:19.261 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:19.256+0000 7fac12a52640 -1 reset not still connected to 0x55749d402680 2024-09-12T06:46:19.262 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:19.264+0000 7fac12a52640 -1 reset not still connected to 0x55749d403ba0 2024-09-12T06:46:19.263 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:19.264+0000 7fac12a52640 -1 reset not still connected to 0x55749d7ca270 2024-09-12T06:46:19.264 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:19.266+0000 7fac12a52640 -1 reset not still connected to 0x55749d7ca8f0 2024-09-12T06:46:19.264 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:19.266+0000 7fac12a52640 -1 reset not still connected to 0x55749e05cf70 2024-09-12T06:46:19.265 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:19.266+0000 7fac12a52640 -1 reset not still connected to 0x5574a2e61d40 2024-09-12T06:46:19.265 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:19.266+0000 7fac12a52640 -1 reset not still connected to 0x5574a2fa7a00 2024-09-12T06:46:19.265 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:19.266+0000 7fac12a52640 -1 reset not still connected to 0x5574a62b6d00 2024-09-12T06:46:19.266 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:19.266+0000 7fac12a52640 -1 reset not still connected to 0x5574bdabf110 2024-09-12T06:46:19.266 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:19.266+0000 7fac12a52640 -1 reset not still connected to 0x5574bdabf1e0 2024-09-12T06:46:19.266 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:19.266+0000 7fac12a52640 -1 reset not still connected to 0x5574bdabfd40 2024-09-12T06:46:19.266 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:19.266+0000 7fac12a52640 -1 reset not still connected to 0x5574cf6f9ba0 2024-09-12T06:46:19.266 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:19.266+0000 7fac12a52640 -1 reset not still connected to 0x5574cfee1c70 2024-09-12T06:46:19.267 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:19.268+0000 7fac12a52640 -1 reset not still connected to 0x5574d1d6da00 2024-09-12T06:46:19.267 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:19.268+0000 7fac12a52640 -1 reset not still connected to 0x5574d5072c30 2024-09-12T06:46:19.304 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:19.305+0000 7fac12a52640 -1 reset not still connected to 0x55749aea6410 2024-09-12T06:46:19.306 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:19.308+0000 7fac12a52640 -1 reset not still connected to 0x55749b0ec1a0 2024-09-12T06:46:19.307 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:19.308+0000 7fac12a52640 -1 reset not still connected to 0x55749b1cb5f0 2024-09-12T06:46:19.313 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:19.314+0000 7fac12a52640 -1 reset not still connected to 0x55749b1cb860 2024-09-12T06:46:19.313 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:19.314+0000 7fac12a52640 -1 reset not still connected to 0x55749b1cbba0 2024-09-12T06:46:19.313 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:19.314+0000 7fac12a52640 -1 reset not still connected to 0x55749b2bf110 2024-09-12T06:46:19.313 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:19.314+0000 7fac12a52640 -1 reset not still connected to 0x55749b2bfee0 2024-09-12T06:46:19.313 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:19.314+0000 7fac12a52640 -1 reset not still connected to 0x55749c9e9380 2024-09-12T06:46:19.314 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:19.314+0000 7fac12a52640 -1 reset not still connected to 0x55749ca4d040 2024-09-12T06:46:19.314 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:19.314+0000 7fac12a52640 -1 reset not still connected to 0x55749d3c4dd0 2024-09-12T06:46:19.314 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:19.314+0000 7fac12a52640 -1 reset not still connected to 0x55749d402410 2024-09-12T06:46:19.314 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:19.314+0000 7fac12a52640 -1 reset not still connected to 0x55749d402680 2024-09-12T06:46:19.314 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:19.314+0000 7fac12a52640 -1 reset not still connected to 0x55749d403ba0 2024-09-12T06:46:19.314 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:19.314+0000 7fac12a52640 -1 reset not still connected to 0x55749d7ca270 2024-09-12T06:46:19.314 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:19.314+0000 7fac12a52640 -1 reset not still connected to 0x55749d7ca8f0 2024-09-12T06:46:19.314 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:19.314+0000 7fac12a52640 -1 reset not still connected to 0x55749e05cf70 2024-09-12T06:46:19.314 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:19.314+0000 7fac12a52640 -1 reset not still connected to 0x5574a2e61d40 2024-09-12T06:46:19.314 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:19.314+0000 7fac12a52640 -1 reset not still connected to 0x5574a2fa7a00 2024-09-12T06:46:19.314 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:19.314+0000 7fac12a52640 -1 reset not still connected to 0x5574a62b6d00 2024-09-12T06:46:19.314 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:19.314+0000 7fac12a52640 -1 reset not still connected to 0x5574b3848d00 2024-09-12T06:46:19.314 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:19.314+0000 7fac12a52640 -1 reset not still connected to 0x5574bdabf110 2024-09-12T06:46:19.314 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:19.314+0000 7fac12a52640 -1 reset not still connected to 0x5574bdabf1e0 2024-09-12T06:46:19.315 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:19.314+0000 7fac12a52640 -1 reset not still connected to 0x5574bdabfd40 2024-09-12T06:46:19.315 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:19.314+0000 7fac12a52640 -1 reset not still connected to 0x5574cf6f9ba0 2024-09-12T06:46:19.315 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:19.314+0000 7fac12a52640 -1 reset not still connected to 0x5574cfee1c70 2024-09-12T06:46:19.315 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:19.314+0000 7fac12a52640 -1 reset not still connected to 0x5574d1d6da00 2024-09-12T06:46:19.315 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:19.314+0000 7fac12a52640 -1 reset not still connected to 0x5574d5072c30 2024-09-12T06:46:20.477 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:20.475+0000 7fa325ae1640 -1 reset not still connected to 0x55f85bac29c0 2024-09-12T06:46:20.477 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:20.475+0000 7fa325ae1640 -1 reset not still connected to 0x55f85bac2b60 2024-09-12T06:46:20.477 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:20.477+0000 7fa325ae1640 -1 reset not still connected to 0x55f85cc67860 2024-09-12T06:46:20.477 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:20.477+0000 7fa325ae1640 -1 reset not still connected to 0x55f85cc67930 2024-09-12T06:46:20.477 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:20.477+0000 7fa325ae1640 -1 reset not still connected to 0x55f85d61d380 2024-09-12T06:46:20.481 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:20.480+0000 7fa325ae1640 -1 reset not still connected to 0x55f8636984e0 2024-09-12T06:46:20.481 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:20.480+0000 7fa325ae1640 -1 reset not still connected to 0x55f863699040 2024-09-12T06:46:20.481 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:20.480+0000 7fa325ae1640 -1 reset not still connected to 0x55f86ee2e340 2024-09-12T06:46:20.481 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:20.480+0000 7fa325ae1640 -1 reset not still connected to 0x55f87faff040 2024-09-12T06:46:20.481 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:20.480+0000 7fa325ae1640 -1 reset not still connected to 0x55f88c6bfa00 2024-09-12T06:46:20.481 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:20.480+0000 7fa325ae1640 -1 reset not still connected to 0x55f8926ce750 2024-09-12T06:46:20.481 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:20.480+0000 7fa325ae1640 -1 reset not still connected to 0x55f8927168f0 2024-09-12T06:46:20.481 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:20.480+0000 7fa325ae1640 -1 reset not still connected to 0x55f892fe4b60 2024-09-12T06:46:21.668 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:21.669+0000 7fac12a52640 -1 reset not still connected to 0x55749b1ca5b0 2024-09-12T06:46:21.671 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:21.671+0000 7fac12a52640 -1 reset not still connected to 0x55749b1cbe10 2024-09-12T06:46:21.671 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:21.671+0000 7fac12a52640 -1 reset not still connected to 0x55749b1cbee0 2024-09-12T06:46:21.671 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:21.671+0000 7fac12a52640 -1 reset not still connected to 0x55749b2be0d0 2024-09-12T06:46:21.671 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:21.671+0000 7fac12a52640 -1 reset not still connected to 0x55749b2be8f0 2024-09-12T06:46:21.672 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:21.671+0000 7fac12a52640 -1 reset not still connected to 0x55749b2bed00 2024-09-12T06:46:21.672 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:21.671+0000 7fac12a52640 -1 reset not still connected to 0x55749b2bedd0 2024-09-12T06:46:21.672 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:21.671+0000 7fac12a52640 -1 reset not still connected to 0x55749b2bf6c0 2024-09-12T06:46:21.672 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:21.671+0000 7fac12a52640 -1 reset not still connected to 0x55749b2bfc70 2024-09-12T06:46:21.672 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:21.671+0000 7fac12a52640 -1 reset not still connected to 0x55749c9a7860 2024-09-12T06:46:21.673 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:21.674+0000 7fac12a52640 -1 reset not still connected to 0x55749ca4d790 2024-09-12T06:46:21.673 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:21.674+0000 7fac12a52640 -1 reset not still connected to 0x55749d3c4750 2024-09-12T06:46:21.673 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:21.674+0000 7fac12a52640 -1 reset not still connected to 0x55749d4025b0 2024-09-12T06:46:21.673 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:21.674+0000 7fac12a52640 -1 reset not still connected to 0x55749e3c4750 2024-09-12T06:46:21.673 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:21.674+0000 7fac12a52640 -1 reset not still connected to 0x55749e3c48f0 2024-09-12T06:46:21.674 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:21.674+0000 7fac12a52640 -1 reset not still connected to 0x55749e3c4d00 2024-09-12T06:46:21.674 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:21.674+0000 7fac12a52640 -1 reset not still connected to 0x5574b3849380 2024-09-12T06:46:21.677 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:21.679+0000 7fac12a52640 -1 reset not still connected to 0x5574b7fd3c70 2024-09-12T06:46:21.677 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:21.679+0000 7fac12a52640 -1 reset not still connected to 0x5574cfee0680 2024-09-12T06:46:22.282 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:22.284+0000 7fa325ae1640 -1 reset not still connected to 0x55f85ac224e0 2024-09-12T06:46:22.282 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:22.284+0000 7fa325ae1640 -1 reset not still connected to 0x55f85ac229c0 2024-09-12T06:46:22.283 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:22.284+0000 7fa325ae1640 -1 reset not still connected to 0x55f85ba92340 2024-09-12T06:46:22.289 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:22.289+0000 7fa325ae1640 -1 reset not still connected to 0x55f85ba925b0 2024-09-12T06:46:22.289 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:22.289+0000 7fa325ae1640 -1 reset not still connected to 0x55f85ba92d00 2024-09-12T06:46:22.289 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:22.289+0000 7fa325ae1640 -1 reset not still connected to 0x55f85bac29c0 2024-09-12T06:46:22.289 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:22.289+0000 7fa325ae1640 -1 reset not still connected to 0x55f85bac2b60 2024-09-12T06:46:22.289 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:22.289+0000 7fa325ae1640 -1 reset not still connected to 0x55f85cc67860 2024-09-12T06:46:22.289 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:22.289+0000 7fa325ae1640 -1 reset not still connected to 0x55f85cc67930 2024-09-12T06:46:22.290 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:22.289+0000 7fa325ae1640 -1 reset not still connected to 0x55f85d61d380 2024-09-12T06:46:22.290 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:22.289+0000 7fa325ae1640 -1 reset not still connected to 0x55f8636984e0 2024-09-12T06:46:22.290 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:22.289+0000 7fa325ae1640 -1 reset not still connected to 0x55f863699040 2024-09-12T06:46:22.290 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:22.289+0000 7fa325ae1640 -1 reset not still connected to 0x55f86ee2e340 2024-09-12T06:46:22.290 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:22.289+0000 7fa325ae1640 -1 reset not still connected to 0x55f87faff040 2024-09-12T06:46:22.290 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:22.289+0000 7fa325ae1640 -1 reset not still connected to 0x55f88c6bfa00 2024-09-12T06:46:22.290 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:22.289+0000 7fa325ae1640 -1 reset not still connected to 0x55f8926ce750 2024-09-12T06:46:22.290 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:22.289+0000 7fa325ae1640 -1 reset not still connected to 0x55f8927168f0 2024-09-12T06:46:22.290 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:22.289+0000 7fa325ae1640 -1 reset not still connected to 0x55f892fe4b60 2024-09-12T06:46:24.808 INFO:tasks.ceph.cluster2.osd.2.smithi152.stderr:2024-09-12T06:46:24.809+0000 7f4789beb640 -1 reset not still connected to 0x56163d519e10 2024-09-12T06:46:24.808 INFO:tasks.ceph.cluster2.osd.2.smithi152.stderr:2024-09-12T06:46:24.810+0000 7f4789beb640 -1 reset not still connected to 0x56163df360d0 2024-09-12T06:46:24.808 INFO:tasks.ceph.cluster2.osd.2.smithi152.stderr:2024-09-12T06:46:24.810+0000 7f4789beb640 -1 reset not still connected to 0x56163df36270 2024-09-12T06:46:24.833 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:24.834+0000 7fac12a52640 -1 reset not still connected to 0x55749b2be410 2024-09-12T06:46:24.846 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:24.848+0000 7fac12a52640 -1 reset not still connected to 0x55749aea6410 2024-09-12T06:46:24.854 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:24.856+0000 7fac12a52640 -1 reset not still connected to 0x55749b1cb860 2024-09-12T06:46:24.855 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:24.856+0000 7fac12a52640 -1 reset not still connected to 0x55749b1cbba0 2024-09-12T06:46:24.855 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:24.856+0000 7fac12a52640 -1 reset not still connected to 0x55749b2bf110 2024-09-12T06:46:24.856 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:24.857+0000 7fac12a52640 -1 reset not still connected to 0x55749ca4d040 2024-09-12T06:46:24.856 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:24.857+0000 7fac12a52640 -1 reset not still connected to 0x55749d3c4dd0 2024-09-12T06:46:24.856 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:24.857+0000 7fac12a52640 -1 reset not still connected to 0x55749d402410 2024-09-12T06:46:24.856 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:24.857+0000 7fac12a52640 -1 reset not still connected to 0x55749d402680 2024-09-12T06:46:24.856 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:24.857+0000 7fac12a52640 -1 reset not still connected to 0x55749d403ba0 2024-09-12T06:46:24.856 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:24.857+0000 7fac12a52640 -1 reset not still connected to 0x55749d7ca270 2024-09-12T06:46:24.857 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:24.858+0000 7fac12a52640 -1 reset not still connected to 0x5574a2e61d40 2024-09-12T06:46:24.857 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:24.858+0000 7fac12a52640 -1 reset not still connected to 0x5574a2fa7a00 2024-09-12T06:46:24.857 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:24.858+0000 7fac12a52640 -1 reset not still connected to 0x5574bdabf110 2024-09-12T06:46:24.857 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:24.858+0000 7fac12a52640 -1 reset not still connected to 0x5574bdabf1e0 2024-09-12T06:46:24.858 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:24.858+0000 7fac12a52640 -1 reset not still connected to 0x5574cf6f9ba0 2024-09-12T06:46:24.858 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:24.858+0000 7fac12a52640 -1 reset not still connected to 0x5574cfee1c70 2024-09-12T06:46:24.858 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:24.858+0000 7fac12a52640 -1 reset not still connected to 0x5574d5072c30 2024-09-12T06:46:24.860 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:24.862+0000 7fac12a52640 -1 reset not still connected to 0x55749b2be410 2024-09-12T06:46:25.057 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:25.051+0000 7fac12a52640 -1 reset not still connected to 0x5574a2fa7a00 2024-09-12T06:46:25.057 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:25.051+0000 7fac12a52640 -1 reset not still connected to 0x5574a62b6d00 2024-09-12T06:46:25.058 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:25.057+0000 7fac12a52640 -1 reset not still connected to 0x5574bdabf110 2024-09-12T06:46:25.058 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:25.057+0000 7fac12a52640 -1 reset not still connected to 0x5574bdabf1e0 2024-09-12T06:46:25.058 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:25.058+0000 7fac12a52640 -1 reset not still connected to 0x5574cf6f9ba0 2024-09-12T06:46:25.058 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:25.058+0000 7fac12a52640 -1 reset not still connected to 0x5574cfee1c70 2024-09-12T06:46:25.059 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:25.060+0000 7fac12a52640 -1 reset not still connected to 0x5574d1d6da00 2024-09-12T06:46:25.059 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:25.060+0000 7fac12a52640 -1 reset not still connected to 0x5574d5072c30 2024-09-12T06:46:25.065 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:25.067+0000 7fac12a52640 -1 reset not still connected to 0x55749aea6410 2024-09-12T06:46:25.067 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:25.068+0000 7fac12a52640 -1 reset not still connected to 0x55749b2bf110 2024-09-12T06:46:25.071 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:25.072+0000 7fac12a52640 -1 reset not still connected to 0x55749b2bfee0 2024-09-12T06:46:25.071 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:25.072+0000 7fac12a52640 -1 reset not still connected to 0x55749c9e9380 2024-09-12T06:46:25.072 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:25.074+0000 7fac12a52640 -1 reset not still connected to 0x55749d402410 2024-09-12T06:46:25.073 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:25.074+0000 7fac12a52640 -1 reset not still connected to 0x55749d402680 2024-09-12T06:46:25.075 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:25.076+0000 7fac12a52640 -1 reset not still connected to 0x55749d403ba0 2024-09-12T06:46:25.077 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:25.076+0000 7fac12a52640 -1 reset not still connected to 0x55749d7ca270 2024-09-12T06:46:25.077 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:25.079+0000 7fac12a52640 -1 reset not still connected to 0x55749d7ca8f0 2024-09-12T06:46:25.078 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:25.079+0000 7fac12a52640 -1 reset not still connected to 0x55749e05cf70 2024-09-12T06:46:25.078 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:25.079+0000 7fac12a52640 -1 reset not still connected to 0x5574a2e61d40 2024-09-12T06:46:25.078 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:25.079+0000 7fac12a52640 -1 reset not still connected to 0x5574a2fa7a00 2024-09-12T06:46:25.078 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:25.079+0000 7fac12a52640 -1 reset not still connected to 0x5574a62b6d00 2024-09-12T06:46:25.079 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:25.079+0000 7fac12a52640 -1 reset not still connected to 0x5574b3848d00 2024-09-12T06:46:25.081 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:25.082+0000 7fac12a52640 -1 reset not still connected to 0x5574bdabf110 2024-09-12T06:46:25.081 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:25.082+0000 7fac12a52640 -1 reset not still connected to 0x5574bdabf1e0 2024-09-12T06:46:25.081 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:25.082+0000 7fac12a52640 -1 reset not still connected to 0x5574bdabfd40 2024-09-12T06:46:25.082 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:25.082+0000 7fac12a52640 -1 reset not still connected to 0x5574cf6f9ba0 2024-09-12T06:46:25.082 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:25.082+0000 7fac12a52640 -1 reset not still connected to 0x5574cfee1c70 2024-09-12T06:46:25.082 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:25.082+0000 7fac12a52640 -1 reset not still connected to 0x5574d1d6da00 2024-09-12T06:46:25.082 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:25.082+0000 7fac12a52640 -1 reset not still connected to 0x5574d5072c30 2024-09-12T06:46:25.218 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:46:25.219+0000 7f80d15bb640 -1 reset not still connected to 0x556b07468000 2024-09-12T06:46:25.218 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:46:25.219+0000 7f80d15bb640 -1 reset not still connected to 0x556b0bcea9c0 2024-09-12T06:46:25.220 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:46:25.222+0000 7f80d15bb640 -1 reset not still connected to 0x556b33ff76c0 2024-09-12T06:46:31.046 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:31.047+0000 7fac12a52640 -1 reset not still connected to 0x55749b1ca5b0 2024-09-12T06:46:31.046 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:31.047+0000 7fac12a52640 -1 reset not still connected to 0x55749b1cbe10 2024-09-12T06:46:31.046 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:31.047+0000 7fac12a52640 -1 reset not still connected to 0x55749b1cbee0 2024-09-12T06:46:31.052 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:31.054+0000 7fac12a52640 -1 reset not still connected to 0x55749b2be0d0 2024-09-12T06:46:31.052 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:31.054+0000 7fac12a52640 -1 reset not still connected to 0x55749b2be8f0 2024-09-12T06:46:31.052 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:31.054+0000 7fac12a52640 -1 reset not still connected to 0x55749b2bed00 2024-09-12T06:46:31.052 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:31.054+0000 7fac12a52640 -1 reset not still connected to 0x55749b2bedd0 2024-09-12T06:46:31.052 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:31.054+0000 7fac12a52640 -1 reset not still connected to 0x55749b2bf6c0 2024-09-12T06:46:31.052 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:31.054+0000 7fac12a52640 -1 reset not still connected to 0x55749b2bfc70 2024-09-12T06:46:31.052 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:31.054+0000 7fac12a52640 -1 reset not still connected to 0x55749c9a7860 2024-09-12T06:46:31.052 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:31.054+0000 7fac12a52640 -1 reset not still connected to 0x55749ca4d790 2024-09-12T06:46:31.052 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:31.054+0000 7fac12a52640 -1 reset not still connected to 0x55749d3c4750 2024-09-12T06:46:31.053 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:31.054+0000 7fac12a52640 -1 reset not still connected to 0x55749d4025b0 2024-09-12T06:46:31.053 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:31.054+0000 7fac12a52640 -1 reset not still connected to 0x55749e3c4750 2024-09-12T06:46:31.053 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:31.054+0000 7fac12a52640 -1 reset not still connected to 0x55749e3c48f0 2024-09-12T06:46:31.053 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:31.054+0000 7fac12a52640 -1 reset not still connected to 0x55749e3c4d00 2024-09-12T06:46:31.053 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:31.054+0000 7fac12a52640 -1 reset not still connected to 0x5574b3849380 2024-09-12T06:46:31.053 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:31.054+0000 7fac12a52640 -1 reset not still connected to 0x5574b7fd3c70 2024-09-12T06:46:31.053 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:46:31.054+0000 7fac12a52640 -1 reset not still connected to 0x5574cfee0680 2024-09-12T06:46:32.354 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:32.353+0000 7fa325ae1640 -1 reset not still connected to 0x55f88b2cd5f0 2024-09-12T06:46:32.360 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:32.361+0000 7fa325ae1640 -1 reset not still connected to 0x55f890e69a00 2024-09-12T06:46:32.362 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:32.363+0000 7fa325ae1640 -1 reset not still connected to 0x55f8926ce0d0 2024-09-12T06:46:32.362 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:32.363+0000 7fa325ae1640 -1 reset not still connected to 0x55f8926ce1a0 2024-09-12T06:46:32.362 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:32.363+0000 7fa325ae1640 -1 reset not still connected to 0x55f896243040 2024-09-12T06:46:38.350 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2024-09-12T06:46:38.350 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep 'image health:' 2024-09-12T06:46:38.350 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut '-d ' -f 3 2024-09-12T06:46:38.388 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: failed to query services: (13) Permission denied 2024-09-12T06:46:38.392 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ state=OK 2024-09-12T06:46:38.392 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test OK = ERROR 2024-09-12T06:46:38.392 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test OK = OK 2024-09-12T06:46:38.393 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:46:38.393 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_pool_images cluster1 mirror 50 2024-09-12T06:46:38.393 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:46:38.393 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:46:38.393 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_count=50 2024-09-12T06:46:38.393 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:46:38.393 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local count 2024-09-12T06:46:38.393 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local last_count=0 2024-09-12T06:46:38.393 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:46:38.393 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 1 40 2024-09-12T06:46:38.394 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in `seq 1 40` 2024-09-12T06:46:38.394 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 1 -ne 1 2024-09-12T06:46:38.394 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-09-12T06:46:38.394 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep 'images: ' 2024-09-12T06:46:38.394 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut '-d ' -f 2 2024-09-12T06:46:38.440 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: failed to query services: (13) Permission denied 2024-09-12T06:46:38.443 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ count=50 2024-09-12T06:46:38.443 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 50 = 50 2024-09-12T06:46:38.444 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:46:38.444 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_pool_healthy cluster1 mirror 2024-09-12T06:46:38.444 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:46:38.444 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:46:38.444 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:46:38.444 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state 2024-09-12T06:46:38.444 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 1 40 2024-09-12T06:46:38.445 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in `seq 1 40` 2024-09-12T06:46:38.445 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 1 -ne 1 2024-09-12T06:46:38.445 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-09-12T06:46:38.445 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep 'image health:' 2024-09-12T06:46:38.445 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut '-d ' -f 3 2024-09-12T06:46:38.489 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: failed to query services: (13) Permission denied 2024-09-12T06:46:38.492 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ state=OK 2024-09-12T06:46:38.492 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test OK = ERROR 2024-09-12T06:46:38.492 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test OK = OK 2024-09-12T06:46:38.492 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:46:38.492 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ testlog 'TEST: compare many images' 2024-09-12T06:46:38.492 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ tee -a /tmp/tmp.GzjHZGs9Z0/rbd-mirror.test.log 2024-09-12T06:46:38.492 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ date '+%F %T' 2024-09-12T06:46:38.502 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo 2024-09-12 06:46:38 TEST: compare many images 2024-09-12T06:46:38.502 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:2024-09-12 06:46:38 TEST: compare many images 2024-09-12T06:46:38.512 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 1 50 2024-09-12T06:46:38.513 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:46:38.513 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_1 2024-09-12T06:46:38.513 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_snap cluster2 mirror image_1 snap 2024-09-12T06:46:38.514 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:46:38.514 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:46:38.514 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_1 2024-09-12T06:46:38.514 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:46:38.514 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror snap create image_1@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T06:46:40.601 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_started cluster1 mirror image_1 2024-09-12T06:46:40.601 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:46:40.601 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:46:40.601 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_1 2024-09-12T06:46:40.601 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_state cluster1 mirror image_1 started 2024-09-12T06:46:40.602 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:46:40.602 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:46:40.602 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_1 2024-09-12T06:46:40.602 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=started 2024-09-12T06:46:40.602 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:46:40.602 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:46:40.602 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:46:41.603 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_replay_state cluster1 mirror image_1 started 2024-09-12T06:46:41.905 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:46:41.905 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:46:41.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_1 2024-09-12T06:46:41.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=started 2024-09-12T06:46:41.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local status_result 2024-09-12T06:46:41.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=stopped 2024-09-12T06:46:41.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_1 2024-09-12T06:46:41.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_instance=cluster1 2024-09-12T06:46:41.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:46:41.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:46:41.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance=cluster1 2024-09-12T06:46:41.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -i state 2024-09-12T06:46:41.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local loop_instance 2024-09-12T06:46:41.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:46:41.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sleep 0 2024-09-12T06:46:41.913 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:46:41.913 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ seq 0 1 2024-09-12T06:46:41.913 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:46:41.913 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_1 2024-09-12T06:46:41.913 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:0 2024-09-12T06:46:41.913 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:46:41.913 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:46:41.914 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:46:41.914 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:46:41.914 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:46:41.914 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:46:41.914 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:46:41.914 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:46:41.914 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:46:41.914 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:46:41.914 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:46:41.914 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:46:41.914 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:46:41.914 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:46:41.914 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:0 2024-09-12T06:46:41.914 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:46:41.915 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:46:41.915 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:46:41.915 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:0 2024-09-12T06:46:41.915 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:46:41.915 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:46:41.915 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:46:41.915 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:46:41.915 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T06:46:41.915 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:46:41.915 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:46:41.915 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=0 2024-09-12T06:46:41.915 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=0 2024-09-12T06:46:41.916 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:46:41.916 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:46:41.916 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:46:41.916 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:46:41.916 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_1 2024-09-12T06:46:41.916 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:no valid command found; 5 closest matches: 2024-09-12T06:46:41.916 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd mirror status mirror/image_11 2024-09-12T06:46:41.916 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd mirror status mirror/image_13 2024-09-12T06:46:41.916 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd mirror status mirror/image_15 2024-09-12T06:46:41.916 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd mirror status mirror/image_17 2024-09-12T06:46:41.916 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd mirror status mirror/image_19 2024-09-12T06:46:41.916 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:admin_socket: invalid command 2024-09-12T06:46:41.916 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:46:41.916 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_1 2024-09-12T06:46:41.917 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:1 2024-09-12T06:46:41.917 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:46:41.917 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:46:41.917 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T06:46:41.917 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:1 2024-09-12T06:46:41.917 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:46:41.917 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:46:41.917 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:46:41.917 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:46:41.917 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T06:46:41.917 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:46:41.917 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:46:41.918 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=1 2024-09-12T06:46:41.918 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:46:41.918 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T06:46:41.918 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:1 2024-09-12T06:46:41.918 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:46:41.918 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:46:41.918 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T06:46:41.918 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:1 2024-09-12T06:46:41.918 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:46:41.918 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:46:41.918 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:46:41.918 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:46:41.918 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T06:46:41.918 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:46:41.919 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:46:41.919 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=1 2024-09-12T06:46:41.919 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=1 2024-09-12T06:46:41.919 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T06:46:41.919 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:46:41.919 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:46:41.919 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:46:41.919 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_1 2024-09-12T06:46:41.936 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ return 0 2024-09-12T06:46:41.937 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T06:46:41.937 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo ' "state": "Replaying"' 2024-09-12T06:46:41.937 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep -i Replaying 2024-09-12T06:46:41.938 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: "state": "Replaying" 2024-09-12T06:46:41.939 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_state=started 2024-09-12T06:46:41.940 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test started = started 2024-09-12T06:46:41.940 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:46:41.940 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_1 2024-09-12T06:46:41.940 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:46:41.940 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:46:41.940 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:46:41.940 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:46:41.940 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_1 2024-09-12T06:46:41.940 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' journal = journal ']' 2024-09-12T06:46:41.940 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror image_1 2024-09-12T06:46:41.940 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:46:41.940 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:46:41.941 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:46:41.941 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:46:41.941 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_1 2024-09-12T06:46:41.941 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-12T06:46:41.941 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-12T06:46:41.941 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:46:41.941 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:46:41.941 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:46:42.141 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror image_1 2024-09-12T06:46:42.141 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:46:42.141 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:46:42.141 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_1 2024-09-12T06:46:42.141 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:46:42.141 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n image_1 ']' 2024-09-12T06:46:42.141 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/image_1' 2024-09-12T06:46:42.141 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_1 2024-09-12T06:46:42.141 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:46:42.142 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:46:42.142 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:46:42.142 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:46:42.142 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:46:42.142 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:46:42.142 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:46:42.142 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:46:42.142 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:46:42.145 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:46:42.146 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_1 2024-09-12T06:46:42.146 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:46:42.146 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:46:42.146 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:46:42.146 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:46:42.146 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:46:42.146 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:46:42.146 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:46:42.146 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:46:42.146 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:46:42.146 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:46:42.146 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:46:42.146 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:46:42.146 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:46:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:46:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:46:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:46:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:46:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:46:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:46:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:46:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:46:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:46:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:46:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:46:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:46:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:46:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:46:42.148 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:46:42.148 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:46:42.148 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:46:42.162 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:46:42.163 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:46:42.163 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:46:42.163 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_1 2024-09-12T06:46:42.249 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:no valid command found; 5 closest matches: 2024-09-12T06:46:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd mirror flush mirror/image_11 2024-09-12T06:46:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd mirror flush mirror/image_13 2024-09-12T06:46:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd mirror flush mirror/image_15 2024-09-12T06:46:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd mirror flush mirror/image_17 2024-09-12T06:46:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd mirror flush mirror/image_19 2024-09-12T06:46:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:admin_socket: invalid command 2024-09-12T06:46:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:46:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_1 2024-09-12T06:46:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:1 2024-09-12T06:46:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:46:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:46:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-12T06:46:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:1 2024-09-12T06:46:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:46:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:46:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:46:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=1 2024-09-12T06:46:42.254 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-12T06:46:42.254 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:46:42.254 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:46:42.254 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=1 2024-09-12T06:46:42.254 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:46:42.254 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-12T06:46:42.254 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:1 2024-09-12T06:46:42.254 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:46:42.254 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:46:42.254 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T06:46:42.254 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:1 2024-09-12T06:46:42.254 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:46:42.254 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:46:42.254 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:46:42.255 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:46:42.255 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T06:46:42.255 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:46:42.255 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:46:42.255 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=1 2024-09-12T06:46:42.255 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:46:42.255 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T06:46:42.272 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:46:42.272 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:46:42.272 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:46:42.272 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_1 2024-09-12T06:46:42.371 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:46:42.371 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror image_1 2024-09-12T06:46:42.371 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:46:42.371 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:46:42.371 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_1 2024-09-12T06:46:42.371 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_1 '' 2024-09-12T06:46:42.371 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:46:42.371 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:46:42.372 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_1 2024-09-12T06:46:42.372 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:46:42.372 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_1.status 2024-09-12T06:46:42.372 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_1.status 2024-09-12T06:46:42.372 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:46:42.386 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_1.status 2024-09-12T06:46:42.388 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_1 2024-09-12T06:46:42.388 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_1.status 2024-09-12T06:46:42.435 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 3 2024-09-12T06:46:42.435 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 3 2024-09-12T06:46:42.435 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:46:42.435 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=13463], [object_number=14, tag_tid=1, entry_tid=13462]]], state=connected] 2024-09-12T06:46:42.435 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=13463], [object_number=14, tag_tid=1, entry_tid=13462]]], state=connected] 2024-09-12T06:46:42.437 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_1.status 2024-09-12T06:46:42.439 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-09-12T06:46:42.440 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror image_1 2024-09-12T06:46:42.440 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:46:42.441 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:46:42.441 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_1 2024-09-12T06:46:42.441 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_1 '..*' 2024-09-12T06:46:42.441 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:46:42.441 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:46:42.441 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_1 2024-09-12T06:46:42.441 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:46:42.441 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_1.status 2024-09-12T06:46:42.441 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_1.status 2024-09-12T06:46:42.441 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:46:42.442 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_1.status 2024-09-12T06:46:42.442 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_1 2024-09-12T06:46:42.442 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_1.status 2024-09-12T06:46:42.488 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 3 2024-09-12T06:46:42.488 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 3 2024-09-12T06:46:42.488 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:46:42.488 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=13463], [object_number=14, tag_tid=1, entry_tid=13462]]], state=connected] 2024-09-12T06:46:42.488 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=13463], [object_number=14, tag_tid=1, entry_tid=13462]]], state=connected] 2024-09-12T06:46:42.491 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_1.status 2024-09-12T06:46:42.493 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-09-12T06:46:42.493 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=1' 2024-09-12T06:46:42.493 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:46:42.493 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_snap_present cluster1 mirror image_1 snap 2024-09-12T06:46:42.494 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:46:42.494 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:46:42.494 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_1 2024-09-12T06:46:42.494 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:46:42.494 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:46:42.494 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T06:46:42.494 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:46:43.494 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 info mirror/image_1@snap 2024-09-12T06:46:43.538 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:rbd image 'image_1': 2024-09-12T06:46:43.538 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: size 128 MiB in 32 objects 2024-09-12T06:46:43.538 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: order 22 (4 MiB objects) 2024-09-12T06:46:43.539 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: snapshot_count: 1 2024-09-12T06:46:43.539 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: id: 10afe1402bfa 2024-09-12T06:46:43.539 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: block_name_prefix: rbd_data.10afe1402bfa 2024-09-12T06:46:43.539 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: format: 2 2024-09-12T06:46:43.539 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: features: layering, exclusive-lock, journaling 2024-09-12T06:46:43.539 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: op_features: 2024-09-12T06:46:43.539 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: flags: 2024-09-12T06:46:43.539 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: create_timestamp: Thu Sep 12 06:38:31 2024 2024-09-12T06:46:43.539 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: access_timestamp: Thu Sep 12 06:38:31 2024 2024-09-12T06:46:43.539 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: modify_timestamp: Thu Sep 12 06:39:32 2024 2024-09-12T06:46:43.539 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: protected: False 2024-09-12T06:46:43.539 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: journal: 10afe1402bfa 2024-09-12T06:46:43.539 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring state: enabled 2024-09-12T06:46:43.539 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring mode: journal 2024-09-12T06:46:43.539 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring global id: 858a7b04-9bf2-47bd-bc9e-5027fefdac4b 2024-09-12T06:46:43.540 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring primary: false 2024-09-12T06:46:43.542 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:46:43.542 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror image_1 snap 2024-09-12T06:46:43.543 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:46:43.543 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_1 2024-09-12T06:46:43.543 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:46:43.543 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:46:43.543 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_1.export 2024-09-12T06:46:43.543 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_1.export 2024-09-12T06:46:43.543 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_1.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_1.export 2024-09-12T06:46:43.543 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export image_1@snap /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_1.export 2024-09-12T06:46:43.698 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:46:43.705 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export image_1@snap /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_1.export 2024-09-12T06:46:43.898 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:46:43.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_1.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_1.export 2024-09-12T06:46:43.962 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_1.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_1.export 2024-09-12T06:46:43.995 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:46:43.995 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:46:43.995 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_2 2024-09-12T06:46:43.995 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_snap cluster2 mirror image_2 snap 2024-09-12T06:46:43.995 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:46:43.995 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:46:43.995 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_2 2024-09-12T06:46:43.995 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:46:43.995 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror snap create image_2@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T06:46:45.608 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_started cluster1 mirror image_2 2024-09-12T06:46:45.608 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:46:45.608 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:46:45.608 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_2 2024-09-12T06:46:45.608 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_state cluster1 mirror image_2 started 2024-09-12T06:46:45.608 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:46:45.609 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:46:45.609 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_2 2024-09-12T06:46:45.609 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=started 2024-09-12T06:46:45.609 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:46:45.609 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:46:45.609 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:46:46.608 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_replay_state cluster1 mirror image_2 started 2024-09-12T06:46:46.608 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:46:46.608 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:46:46.609 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_2 2024-09-12T06:46:46.609 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=started 2024-09-12T06:46:46.609 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local status_result 2024-09-12T06:46:46.609 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=stopped 2024-09-12T06:46:46.610 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_2 2024-09-12T06:46:46.610 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_instance=cluster1 2024-09-12T06:46:46.610 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:46:46.610 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:46:46.610 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -i state 2024-09-12T06:46:46.610 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance=cluster1 2024-09-12T06:46:46.610 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local loop_instance 2024-09-12T06:46:46.610 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:46:46.610 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sleep 0 2024-09-12T06:46:46.613 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:46:46.613 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ seq 0 1 2024-09-12T06:46:46.613 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:46:46.613 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_2 2024-09-12T06:46:46.614 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:0 2024-09-12T06:46:46.614 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:46:46.614 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:46:46.614 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:46:46.614 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:46:46.614 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:46:46.614 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:46:46.614 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:46:46.614 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:46:46.614 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:46:46.614 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:46:46.614 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:46:46.614 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:46:46.614 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:46:46.614 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:46:46.615 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:0 2024-09-12T06:46:46.615 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:46:46.615 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:46:46.615 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:46:46.615 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:0 2024-09-12T06:46:46.615 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:46:46.615 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:46:46.615 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:46:46.615 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:46:46.615 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T06:46:46.616 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:46:46.616 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:46:46.616 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=0 2024-09-12T06:46:46.616 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=0 2024-09-12T06:46:46.616 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:46:46.632 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:46:46.632 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:46:46.633 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:46:46.633 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_2 2024-09-12T06:46:46.735 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ return 0 2024-09-12T06:46:46.736 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T06:46:46.736 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo ' "state": "Replaying"' 2024-09-12T06:46:46.736 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep -i Replaying 2024-09-12T06:46:46.737 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: "state": "Replaying" 2024-09-12T06:46:46.738 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_state=started 2024-09-12T06:46:46.738 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test started = started 2024-09-12T06:46:46.739 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:46:46.739 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_2 2024-09-12T06:46:46.739 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:46:46.739 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:46:46.739 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:46:46.739 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:46:46.739 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_2 2024-09-12T06:46:46.739 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' journal = journal ']' 2024-09-12T06:46:46.739 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror image_2 2024-09-12T06:46:46.739 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:46:46.739 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:46:46.739 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:46:46.739 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:46:46.739 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_2 2024-09-12T06:46:46.740 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-12T06:46:46.740 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-12T06:46:46.740 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:46:46.740 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:46:46.740 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:46:46.939 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror image_2 2024-09-12T06:46:46.940 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:46:46.940 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:46:46.940 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_2 2024-09-12T06:46:46.940 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:46:46.940 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n image_2 ']' 2024-09-12T06:46:46.940 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/image_2' 2024-09-12T06:46:46.940 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_2 2024-09-12T06:46:46.940 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:46:46.940 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:46:46.940 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:46:46.940 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:46:46.940 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:46:46.941 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:46:46.941 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:46:46.941 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:46:46.941 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:46:46.944 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:46:46.944 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_2 2024-09-12T06:46:46.944 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:46:46.944 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:46:46.944 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:46:46.944 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:46:46.944 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:46:46.944 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:46:46.944 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:46:46.944 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:46:46.944 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:46:46.945 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:46:46.945 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:46:46.945 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:46:46.945 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:46:46.945 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:46:46.945 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:46:46.945 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:46:46.945 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:46:46.945 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:46:46.945 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:46:46.945 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:46:46.945 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:46:46.945 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:46:46.945 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:46:46.946 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:46:46.946 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:46:46.946 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:46:46.946 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:46:46.946 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:46:46.946 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:46:46.946 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:46:46.962 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:46:46.962 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:46:46.962 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:46:46.962 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_2 2024-09-12T06:46:47.061 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:46:47.061 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror image_2 2024-09-12T06:46:47.061 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:46:47.061 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:46:47.061 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_2 2024-09-12T06:46:47.061 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_2 '' 2024-09-12T06:46:47.062 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:46:47.062 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:46:47.062 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_2 2024-09-12T06:46:47.062 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:46:47.062 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_2.status 2024-09-12T06:46:47.062 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_2.status 2024-09-12T06:46:47.062 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:46:47.062 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_2.status 2024-09-12T06:46:47.063 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_2 2024-09-12T06:46:47.063 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_2.status 2024-09-12T06:46:47.109 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 2 2024-09-12T06:46:47.109 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 2 2024-09-12T06:46:47.109 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:46:47.109 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=7394], [object_number=11, tag_tid=1, entry_tid=7391]]], state=connected] 2024-09-12T06:46:47.109 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=7394], [object_number=11, tag_tid=1, entry_tid=7391]]], state=connected] 2024-09-12T06:46:47.111 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_2.status 2024-09-12T06:46:47.113 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-09-12T06:46:47.115 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror image_2 2024-09-12T06:46:47.115 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:46:47.115 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:46:47.115 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_2 2024-09-12T06:46:47.115 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_2 '..*' 2024-09-12T06:46:47.115 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:46:47.116 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:46:47.116 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_2 2024-09-12T06:46:47.116 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:46:47.116 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_2.status 2024-09-12T06:46:47.116 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_2.status 2024-09-12T06:46:47.116 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:46:47.116 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_2.status 2024-09-12T06:46:47.116 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_2 2024-09-12T06:46:47.117 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_2.status 2024-09-12T06:46:47.161 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 2 2024-09-12T06:46:47.161 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 2 2024-09-12T06:46:47.161 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:46:47.161 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=7394], [object_number=11, tag_tid=1, entry_tid=7391]]], state=connected] 2024-09-12T06:46:47.161 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=7394], [object_number=11, tag_tid=1, entry_tid=7391]]], state=connected] 2024-09-12T06:46:47.164 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_2.status 2024-09-12T06:46:47.166 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-09-12T06:46:47.166 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-09-12T06:46:47.167 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:46:47.167 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_snap_present cluster1 mirror image_2 snap 2024-09-12T06:46:47.167 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:46:47.167 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:46:47.167 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_2 2024-09-12T06:46:47.167 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:46:47.167 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:46:47.167 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T06:46:47.167 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:46:48.167 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 info mirror/image_2@snap 2024-09-12T06:46:48.211 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:rbd image 'image_2': 2024-09-12T06:46:48.212 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: size 128 MiB in 32 objects 2024-09-12T06:46:48.212 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: order 22 (4 MiB objects) 2024-09-12T06:46:48.212 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: snapshot_count: 1 2024-09-12T06:46:48.212 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: id: 10a8fed7b3cf 2024-09-12T06:46:48.212 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: block_name_prefix: rbd_data.10a8fed7b3cf 2024-09-12T06:46:48.212 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: format: 2 2024-09-12T06:46:48.212 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: features: layering, exclusive-lock, journaling 2024-09-12T06:46:48.212 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: op_features: 2024-09-12T06:46:48.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: flags: 2024-09-12T06:46:48.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: create_timestamp: Thu Sep 12 06:38:42 2024 2024-09-12T06:46:48.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: access_timestamp: Thu Sep 12 06:38:42 2024 2024-09-12T06:46:48.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: modify_timestamp: Thu Sep 12 06:38:42 2024 2024-09-12T06:46:48.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: protected: False 2024-09-12T06:46:48.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: journal: 10a8fed7b3cf 2024-09-12T06:46:48.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring state: enabled 2024-09-12T06:46:48.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring mode: journal 2024-09-12T06:46:48.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring global id: c0594577-465d-4756-93a7-f5e4463b2e29 2024-09-12T06:46:48.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring primary: false 2024-09-12T06:46:48.215 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:46:48.216 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror image_2 snap 2024-09-12T06:46:48.216 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:46:48.216 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_2 2024-09-12T06:46:48.216 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:46:48.216 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:46:48.216 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_2.export 2024-09-12T06:46:48.216 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_2.export 2024-09-12T06:46:48.216 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_2.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_2.export 2024-09-12T06:46:48.216 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export image_2@snap /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_2.export 2024-09-12T06:46:48.358 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:46:48.365 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export image_2@snap /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_2.export 2024-09-12T06:46:48.571 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:46:48.576 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_2.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_2.export 2024-09-12T06:46:48.631 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_2.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_2.export 2024-09-12T06:46:48.665 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:46:48.665 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:46:48.665 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_3 2024-09-12T06:46:48.665 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_snap cluster2 mirror image_3 snap 2024-09-12T06:46:48.665 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:46:48.665 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:46:48.665 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_3 2024-09-12T06:46:48.666 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:46:48.666 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror snap create image_3@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T06:46:51.628 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_started cluster1 mirror image_3 2024-09-12T06:46:51.629 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:46:51.629 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:46:51.629 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_3 2024-09-12T06:46:51.629 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_state cluster1 mirror image_3 started 2024-09-12T06:46:51.629 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:46:51.629 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:46:51.629 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_3 2024-09-12T06:46:51.629 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=started 2024-09-12T06:46:51.629 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:46:51.629 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:46:51.629 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:46:52.630 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_replay_state cluster1 mirror image_3 started 2024-09-12T06:46:52.630 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:46:52.630 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:46:52.630 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_3 2024-09-12T06:46:52.630 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=started 2024-09-12T06:46:52.630 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local status_result 2024-09-12T06:46:52.630 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=stopped 2024-09-12T06:46:52.630 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_3 2024-09-12T06:46:52.630 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_instance=cluster1 2024-09-12T06:46:52.630 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:46:52.631 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:46:52.631 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -i state 2024-09-12T06:46:52.631 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance=cluster1 2024-09-12T06:46:52.631 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local loop_instance 2024-09-12T06:46:52.631 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:46:52.631 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sleep 0 2024-09-12T06:46:52.633 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:46:52.633 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ seq 0 1 2024-09-12T06:46:52.633 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:46:52.633 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_3 2024-09-12T06:46:52.633 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:0 2024-09-12T06:46:52.633 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:46:52.633 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:46:52.633 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:46:52.634 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:46:52.634 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:46:52.634 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:46:52.634 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:46:52.634 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:46:52.634 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:46:52.634 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:46:52.634 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:46:52.634 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:46:52.634 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:46:52.634 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:46:52.634 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:0 2024-09-12T06:46:52.636 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:46:52.636 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:46:52.636 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:46:52.636 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:0 2024-09-12T06:46:52.636 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:46:52.636 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:46:52.637 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:46:52.637 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:46:52.637 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T06:46:52.637 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:46:52.637 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:46:52.637 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=0 2024-09-12T06:46:52.637 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=0 2024-09-12T06:46:52.637 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:46:52.654 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:46:52.654 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:46:52.654 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:46:52.654 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_3 2024-09-12T06:46:52.755 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ return 0 2024-09-12T06:46:52.755 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T06:46:52.756 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo ' "state": "Replaying"' 2024-09-12T06:46:52.756 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep -i Replaying 2024-09-12T06:46:52.756 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: "state": "Replaying" 2024-09-12T06:46:52.758 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_state=started 2024-09-12T06:46:52.758 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test started = started 2024-09-12T06:46:52.758 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:46:52.758 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_3 2024-09-12T06:46:52.758 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:46:52.758 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:46:52.758 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:46:52.759 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:46:52.759 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_3 2024-09-12T06:46:52.759 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' journal = journal ']' 2024-09-12T06:46:52.759 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror image_3 2024-09-12T06:46:52.759 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:46:52.759 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:46:52.759 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:46:52.759 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:46:52.759 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_3 2024-09-12T06:46:52.759 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-12T06:46:52.759 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-12T06:46:52.759 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:46:52.759 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:46:52.759 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:46:52.959 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror image_3 2024-09-12T06:46:52.959 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:46:52.959 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:46:52.959 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_3 2024-09-12T06:46:52.959 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:46:52.959 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n image_3 ']' 2024-09-12T06:46:52.959 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/image_3' 2024-09-12T06:46:52.960 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_3 2024-09-12T06:46:52.960 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:46:52.960 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:46:52.960 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:46:52.960 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:46:52.960 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:46:52.960 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:46:52.960 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:46:52.960 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:46:52.960 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:46:52.963 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:46:52.964 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_3 2024-09-12T06:46:52.964 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:46:52.964 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:46:52.964 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:46:52.964 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:46:52.964 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:46:52.964 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:46:52.964 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:46:52.964 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:46:52.964 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:46:52.964 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:46:52.964 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:46:52.964 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:46:52.965 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:46:52.965 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:46:52.965 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:46:52.965 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:46:52.965 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:46:52.965 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:46:52.965 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:46:52.965 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:46:52.965 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:46:52.965 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:46:52.965 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:46:52.965 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:46:52.965 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:46:52.965 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:46:52.966 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:46:52.966 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:46:52.966 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:46:52.966 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:46:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:46:52.981 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:46:52.981 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:46:52.981 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_3 2024-09-12T06:46:53.080 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:46:53.080 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror image_3 2024-09-12T06:46:53.080 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:46:53.080 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:46:53.081 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_3 2024-09-12T06:46:53.081 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_3 '' 2024-09-12T06:46:53.081 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:46:53.081 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:46:53.081 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_3 2024-09-12T06:46:53.081 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:46:53.081 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_3.status 2024-09-12T06:46:53.081 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_3.status 2024-09-12T06:46:53.081 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:46:53.082 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_3.status 2024-09-12T06:46:53.082 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_3 2024-09-12T06:46:53.082 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_3.status 2024-09-12T06:46:53.127 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 3 2024-09-12T06:46:53.127 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 3 2024-09-12T06:46:53.127 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:46:53.127 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=13463], [object_number=14, tag_tid=1, entry_tid=13462]]], state=connected] 2024-09-12T06:46:53.128 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=13463], [object_number=14, tag_tid=1, entry_tid=13462]]], state=connected] 2024-09-12T06:46:53.130 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_3.status 2024-09-12T06:46:53.131 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-09-12T06:46:53.133 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror image_3 2024-09-12T06:46:53.133 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:46:53.133 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:46:53.133 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_3 2024-09-12T06:46:53.133 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_3 '..*' 2024-09-12T06:46:53.133 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:46:53.133 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:46:53.133 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_3 2024-09-12T06:46:53.133 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:46:53.133 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_3.status 2024-09-12T06:46:53.133 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_3.status 2024-09-12T06:46:53.133 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:46:53.134 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_3.status 2024-09-12T06:46:53.134 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_3 2024-09-12T06:46:53.134 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_3.status 2024-09-12T06:46:53.180 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 3 2024-09-12T06:46:53.180 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 3 2024-09-12T06:46:53.180 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:46:53.180 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=13463], [object_number=14, tag_tid=1, entry_tid=13462]]], state=connected] 2024-09-12T06:46:53.181 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=13463], [object_number=14, tag_tid=1, entry_tid=13462]]], state=connected] 2024-09-12T06:46:53.183 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_3.status 2024-09-12T06:46:53.185 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-09-12T06:46:53.185 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=1' 2024-09-12T06:46:53.185 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:46:53.185 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_snap_present cluster1 mirror image_3 snap 2024-09-12T06:46:53.185 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:46:53.185 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:46:53.186 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_3 2024-09-12T06:46:53.186 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:46:53.186 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:46:53.186 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T06:46:53.186 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:46:54.186 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 info mirror/image_3@snap 2024-09-12T06:46:54.236 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:rbd image 'image_3': 2024-09-12T06:46:54.236 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: size 128 MiB in 32 objects 2024-09-12T06:46:54.236 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: order 22 (4 MiB objects) 2024-09-12T06:46:54.236 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: snapshot_count: 1 2024-09-12T06:46:54.236 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: id: 10a8b6eaeb2f 2024-09-12T06:46:54.236 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: block_name_prefix: rbd_data.10a8b6eaeb2f 2024-09-12T06:46:54.236 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: format: 2 2024-09-12T06:46:54.236 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: features: layering, exclusive-lock, journaling 2024-09-12T06:46:54.236 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: op_features: 2024-09-12T06:46:54.236 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: flags: 2024-09-12T06:46:54.237 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: create_timestamp: Thu Sep 12 06:38:49 2024 2024-09-12T06:46:54.237 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: access_timestamp: Thu Sep 12 06:38:49 2024 2024-09-12T06:46:54.237 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: modify_timestamp: Thu Sep 12 06:39:50 2024 2024-09-12T06:46:54.237 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: protected: False 2024-09-12T06:46:54.237 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: journal: 10a8b6eaeb2f 2024-09-12T06:46:54.237 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring state: enabled 2024-09-12T06:46:54.237 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring mode: journal 2024-09-12T06:46:54.237 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring global id: c99a5356-fcaf-4414-aec1-803561906a85 2024-09-12T06:46:54.237 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring primary: false 2024-09-12T06:46:54.240 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:46:54.240 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror image_3 snap 2024-09-12T06:46:54.240 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:46:54.240 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_3 2024-09-12T06:46:54.240 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:46:54.240 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:46:54.240 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_3.export 2024-09-12T06:46:54.240 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_3.export 2024-09-12T06:46:54.240 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_3.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_3.export 2024-09-12T06:46:54.241 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export image_3@snap /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_3.export 2024-09-12T06:46:54.402 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:46:54.409 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export image_3@snap /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_3.export 2024-09-12T06:46:54.553 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:46:54.555+0000 7fa3176b1640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-09-12T06:46:54.641 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:46:54.649 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_3.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_3.export 2024-09-12T06:46:54.706 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_3.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_3.export 2024-09-12T06:46:54.738 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:46:54.739 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:46:54.739 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_4 2024-09-12T06:46:54.739 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_snap cluster2 mirror image_4 snap 2024-09-12T06:46:54.739 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:46:54.739 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:46:54.739 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_4 2024-09-12T06:46:54.739 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:46:54.739 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror snap create image_4@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T06:46:54.787 INFO:tasks.ceph.cluster2.osd.2.smithi152.stderr:2024-09-12T06:46:54.789+0000 7f4789beb640 -1 reset not still connected to 0x56163d5191e0 2024-09-12T06:46:54.789 INFO:tasks.ceph.cluster2.osd.2.smithi152.stderr:2024-09-12T06:46:54.790+0000 7f4789beb640 -1 reset not still connected to 0x56163e3b3d40 2024-09-12T06:46:56.637 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_started cluster1 mirror image_4 2024-09-12T06:46:56.637 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:46:56.637 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:46:56.637 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_4 2024-09-12T06:46:56.637 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_state cluster1 mirror image_4 started 2024-09-12T06:46:56.637 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:46:56.637 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:46:56.638 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_4 2024-09-12T06:46:56.638 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=started 2024-09-12T06:46:56.638 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:46:56.638 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:46:56.638 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:46:57.638 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_replay_state cluster1 mirror image_4 started 2024-09-12T06:46:57.638 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:46:57.638 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:46:57.639 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_4 2024-09-12T06:46:57.639 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=started 2024-09-12T06:46:57.639 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local status_result 2024-09-12T06:46:57.639 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=stopped 2024-09-12T06:46:57.639 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_4 2024-09-12T06:46:57.639 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_instance=cluster1 2024-09-12T06:46:57.639 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:46:57.639 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:46:57.639 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance=cluster1 2024-09-12T06:46:57.639 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -i state 2024-09-12T06:46:57.639 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local loop_instance 2024-09-12T06:46:57.639 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:46:57.639 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sleep 0 2024-09-12T06:46:57.642 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:46:57.642 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ seq 0 1 2024-09-12T06:46:57.642 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:46:57.642 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_4 2024-09-12T06:46:57.642 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:0 2024-09-12T06:46:57.642 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:46:57.642 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:46:57.642 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:46:57.642 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:46:57.642 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:46:57.642 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:46:57.643 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:46:57.643 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:46:57.643 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:46:57.643 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:46:57.643 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:46:57.643 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:46:57.643 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:46:57.643 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:46:57.643 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:0 2024-09-12T06:46:57.643 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:46:57.643 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:46:57.643 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:46:57.643 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:0 2024-09-12T06:46:57.643 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:46:57.645 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:46:57.645 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:46:57.645 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:46:57.645 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T06:46:57.645 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:46:57.645 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:46:57.645 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=0 2024-09-12T06:46:57.645 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=0 2024-09-12T06:46:57.645 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:46:57.662 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:46:57.662 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:46:57.662 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:46:57.662 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_4 2024-09-12T06:46:57.756 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:no valid command found; 5 closest matches: 2024-09-12T06:46:57.756 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd mirror status mirror/image_41 2024-09-12T06:46:57.756 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd mirror status mirror/image_43 2024-09-12T06:46:57.756 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd mirror status mirror/image_45 2024-09-12T06:46:57.756 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd mirror status mirror/image_47 2024-09-12T06:46:57.756 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd mirror status mirror/image_49 2024-09-12T06:46:57.756 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:admin_socket: invalid command 2024-09-12T06:46:57.759 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:46:57.760 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_4 2024-09-12T06:46:57.760 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:1 2024-09-12T06:46:57.760 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:46:57.760 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:46:57.760 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T06:46:57.760 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:1 2024-09-12T06:46:57.760 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:46:57.760 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:46:57.760 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:46:57.760 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:46:57.760 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T06:46:57.760 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:46:57.760 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:46:57.761 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=1 2024-09-12T06:46:57.761 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:46:57.761 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T06:46:57.761 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:1 2024-09-12T06:46:57.761 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:46:57.761 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:46:57.761 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T06:46:57.761 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:1 2024-09-12T06:46:57.761 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:46:57.761 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:46:57.761 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:46:57.761 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:46:57.761 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T06:46:57.761 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:46:57.762 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:46:57.762 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=1 2024-09-12T06:46:57.762 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=1 2024-09-12T06:46:57.762 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T06:46:57.777 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:46:57.778 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:46:57.778 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:46:57.778 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_4 2024-09-12T06:46:57.878 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ return 0 2024-09-12T06:46:57.879 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T06:46:57.879 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo ' "state": "Replaying"' 2024-09-12T06:46:57.880 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep -i Replaying 2024-09-12T06:46:57.880 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: "state": "Replaying" 2024-09-12T06:46:57.882 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_state=started 2024-09-12T06:46:57.882 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test started = started 2024-09-12T06:46:57.882 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:46:57.882 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_4 2024-09-12T06:46:57.882 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:46:57.882 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:46:57.882 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:46:57.882 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:46:57.882 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_4 2024-09-12T06:46:57.882 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' journal = journal ']' 2024-09-12T06:46:57.882 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror image_4 2024-09-12T06:46:57.882 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:46:57.882 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:46:57.882 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:46:57.883 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:46:57.883 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_4 2024-09-12T06:46:57.883 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-12T06:46:57.883 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-12T06:46:57.883 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:46:57.883 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:46:57.883 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:46:58.086 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror image_4 2024-09-12T06:46:58.086 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:46:58.087 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:46:58.087 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_4 2024-09-12T06:46:58.087 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:46:58.087 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n image_4 ']' 2024-09-12T06:46:58.087 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/image_4' 2024-09-12T06:46:58.087 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_4 2024-09-12T06:46:58.087 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:46:58.087 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:46:58.087 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:46:58.088 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:46:58.088 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:46:58.088 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:46:58.088 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:46:58.088 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:46:58.088 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:46:58.088 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:46:58.088 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_4 2024-09-12T06:46:58.088 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:46:58.088 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:46:58.088 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:46:58.088 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:46:58.089 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:46:58.089 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:46:58.089 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:46:58.089 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:46:58.089 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:46:58.089 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:46:58.089 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:46:58.089 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:46:58.089 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:46:58.089 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:46:58.089 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:46:58.089 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:46:58.089 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:46:58.090 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:46:58.090 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:46:58.090 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:46:58.090 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:46:58.090 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:46:58.090 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:46:58.090 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:46:58.090 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:46:58.090 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:46:58.090 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:46:58.090 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:46:58.090 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:46:58.091 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:46:58.105 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:46:58.105 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:46:58.105 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:46:58.105 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_4 2024-09-12T06:46:58.191 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:no valid command found; 5 closest matches: 2024-09-12T06:46:58.191 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd mirror flush mirror/image_41 2024-09-12T06:46:58.191 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd mirror flush mirror/image_43 2024-09-12T06:46:58.191 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd mirror flush mirror/image_45 2024-09-12T06:46:58.191 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd mirror flush mirror/image_47 2024-09-12T06:46:58.191 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd mirror flush mirror/image_49 2024-09-12T06:46:58.191 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:admin_socket: invalid command 2024-09-12T06:46:58.195 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:46:58.195 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_4 2024-09-12T06:46:58.195 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:1 2024-09-12T06:46:58.195 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:46:58.195 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:46:58.195 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-12T06:46:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:1 2024-09-12T06:46:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:46:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:46:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:46:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=1 2024-09-12T06:46:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-12T06:46:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:46:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:46:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=1 2024-09-12T06:46:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:46:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-12T06:46:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:1 2024-09-12T06:46:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:46:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:46:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T06:46:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:1 2024-09-12T06:46:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:46:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:46:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:46:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:46:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T06:46:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:46:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:46:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=1 2024-09-12T06:46:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:46:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T06:46:58.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:46:58.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:46:58.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:46:58.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_4 2024-09-12T06:46:58.310 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:46:58.310 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror image_4 2024-09-12T06:46:58.310 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:46:58.310 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:46:58.310 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_4 2024-09-12T06:46:58.310 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_4 '' 2024-09-12T06:46:58.310 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:46:58.310 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:46:58.311 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_4 2024-09-12T06:46:58.311 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:46:58.311 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_4.status 2024-09-12T06:46:58.311 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_4.status 2024-09-12T06:46:58.311 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:46:58.311 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_4.status 2024-09-12T06:46:58.311 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_4 2024-09-12T06:46:58.312 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_4.status 2024-09-12T06:46:58.357 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 2 2024-09-12T06:46:58.357 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 2 2024-09-12T06:46:58.357 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:46:58.357 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9203], [object_number=10, tag_tid=1, entry_tid=9202]]], state=connected] 2024-09-12T06:46:58.357 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9203], [object_number=10, tag_tid=1, entry_tid=9202]]], state=connected] 2024-09-12T06:46:58.360 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_4.status 2024-09-12T06:46:58.361 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-09-12T06:46:58.362 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror image_4 2024-09-12T06:46:58.363 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:46:58.363 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:46:58.363 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_4 2024-09-12T06:46:58.363 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_4 '..*' 2024-09-12T06:46:58.363 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:46:58.363 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:46:58.363 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_4 2024-09-12T06:46:58.363 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:46:58.363 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_4.status 2024-09-12T06:46:58.363 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_4.status 2024-09-12T06:46:58.363 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:46:58.364 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_4.status 2024-09-12T06:46:58.364 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_4 2024-09-12T06:46:58.364 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_4.status 2024-09-12T06:46:58.409 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 2 2024-09-12T06:46:58.409 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 2 2024-09-12T06:46:58.409 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:46:58.409 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9203], [object_number=10, tag_tid=1, entry_tid=9202]]], state=connected] 2024-09-12T06:46:58.409 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9203], [object_number=10, tag_tid=1, entry_tid=9202]]], state=connected] 2024-09-12T06:46:58.412 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_4.status 2024-09-12T06:46:58.414 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-09-12T06:46:58.415 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-09-12T06:46:58.415 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:46:58.415 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_snap_present cluster1 mirror image_4 snap 2024-09-12T06:46:58.415 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:46:58.415 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:46:58.415 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_4 2024-09-12T06:46:58.415 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:46:58.415 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:46:58.415 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T06:46:58.415 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:46:59.415 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 info mirror/image_4@snap 2024-09-12T06:46:59.462 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:rbd image 'image_4': 2024-09-12T06:46:59.462 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: size 128 MiB in 32 objects 2024-09-12T06:46:59.462 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: order 22 (4 MiB objects) 2024-09-12T06:46:59.462 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: snapshot_count: 1 2024-09-12T06:46:59.462 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: id: 10af527afda4 2024-09-12T06:46:59.462 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: block_name_prefix: rbd_data.10af527afda4 2024-09-12T06:46:59.463 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: format: 2 2024-09-12T06:46:59.463 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: features: layering, exclusive-lock, journaling 2024-09-12T06:46:59.463 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: op_features: 2024-09-12T06:46:59.463 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: flags: 2024-09-12T06:46:59.463 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: create_timestamp: Thu Sep 12 06:39:01 2024 2024-09-12T06:46:59.463 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: access_timestamp: Thu Sep 12 06:39:01 2024 2024-09-12T06:46:59.463 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: modify_timestamp: Thu Sep 12 06:39:01 2024 2024-09-12T06:46:59.463 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: protected: False 2024-09-12T06:46:59.463 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: journal: 10af527afda4 2024-09-12T06:46:59.463 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring state: enabled 2024-09-12T06:46:59.463 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring mode: journal 2024-09-12T06:46:59.463 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring global id: 6b724db9-c7c4-4ecc-83c7-a291e4c49324 2024-09-12T06:46:59.463 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring primary: false 2024-09-12T06:46:59.466 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:46:59.466 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror image_4 snap 2024-09-12T06:46:59.466 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:46:59.466 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_4 2024-09-12T06:46:59.466 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:46:59.466 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:46:59.466 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_4.export 2024-09-12T06:46:59.466 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_4.export 2024-09-12T06:46:59.466 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_4.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_4.export 2024-09-12T06:46:59.467 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export image_4@snap /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_4.export 2024-09-12T06:46:59.614 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:46:59.621 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export image_4@snap /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_4.export 2024-09-12T06:46:59.845 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:46:59.851 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_4.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_4.export 2024-09-12T06:46:59.907 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_4.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_4.export 2024-09-12T06:46:59.942 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:46:59.942 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:46:59.942 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_5 2024-09-12T06:46:59.942 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_snap cluster2 mirror image_5 snap 2024-09-12T06:46:59.942 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:46:59.942 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:46:59.942 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_5 2024-09-12T06:46:59.943 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:46:59.943 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror snap create image_5@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T06:47:03.664 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_started cluster1 mirror image_5 2024-09-12T06:47:03.665 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:47:03.665 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:47:03.665 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_5 2024-09-12T06:47:03.665 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_state cluster1 mirror image_5 started 2024-09-12T06:47:03.665 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:47:03.665 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:47:03.666 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_5 2024-09-12T06:47:03.666 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=started 2024-09-12T06:47:03.666 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:47:03.666 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:47:03.666 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:47:04.664 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_replay_state cluster1 mirror image_5 started 2024-09-12T06:47:04.665 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:47:04.665 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:47:04.665 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_5 2024-09-12T06:47:04.665 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=started 2024-09-12T06:47:04.665 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local status_result 2024-09-12T06:47:04.665 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=stopped 2024-09-12T06:47:04.669 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_5 2024-09-12T06:47:04.669 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_instance=cluster1 2024-09-12T06:47:04.669 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:47:04.669 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:47:04.669 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance=cluster1 2024-09-12T06:47:04.669 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -i state 2024-09-12T06:47:04.669 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local loop_instance 2024-09-12T06:47:04.669 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:47:04.669 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sleep 0 2024-09-12T06:47:04.670 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:47:04.670 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ seq 0 1 2024-09-12T06:47:04.670 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:47:04.670 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_5 2024-09-12T06:47:04.670 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:0 2024-09-12T06:47:04.670 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:47:04.670 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:47:04.670 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:47:04.670 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:47:04.670 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:47:04.670 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:47:04.670 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:47:04.670 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:47:04.670 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:47:04.671 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:47:04.671 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:47:04.671 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:47:04.671 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:47:04.671 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:47:04.671 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:0 2024-09-12T06:47:04.671 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:47:04.671 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:47:04.671 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:47:04.671 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:0 2024-09-12T06:47:04.671 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:47:04.671 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:47:04.671 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:47:04.671 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:47:04.672 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T06:47:04.672 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:47:04.672 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:47:04.672 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=0 2024-09-12T06:47:04.672 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=0 2024-09-12T06:47:04.672 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:47:04.689 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:47:04.689 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:47:04.689 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:47:04.689 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_5 2024-09-12T06:47:04.806 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ return 0 2024-09-12T06:47:04.807 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T06:47:04.807 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo ' "state": "Replaying"' 2024-09-12T06:47:04.807 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep -i Replaying 2024-09-12T06:47:04.808 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: "state": "Replaying" 2024-09-12T06:47:04.810 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_state=started 2024-09-12T06:47:04.810 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test started = started 2024-09-12T06:47:04.810 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:47:04.810 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_5 2024-09-12T06:47:04.810 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:47:04.810 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:47:04.810 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:47:04.810 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:47:04.810 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_5 2024-09-12T06:47:04.810 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' journal = journal ']' 2024-09-12T06:47:04.810 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror image_5 2024-09-12T06:47:04.810 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:47:04.811 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:47:04.811 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:47:04.811 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:47:04.811 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_5 2024-09-12T06:47:04.811 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-12T06:47:04.811 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-12T06:47:04.811 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:47:04.811 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:47:04.811 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:47:05.011 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror image_5 2024-09-12T06:47:05.011 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:47:05.011 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:47:05.011 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_5 2024-09-12T06:47:05.012 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:47:05.012 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n image_5 ']' 2024-09-12T06:47:05.012 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/image_5' 2024-09-12T06:47:05.012 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_5 2024-09-12T06:47:05.012 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:47:05.012 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:47:05.012 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:47:05.012 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:47:05.012 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:47:05.012 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:47:05.012 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:47:05.012 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:47:05.012 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:47:05.015 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:47:05.015 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_5 2024-09-12T06:47:05.015 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:47:05.015 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:47:05.016 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:47:05.016 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:47:05.016 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:47:05.016 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:47:05.016 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:47:05.016 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:47:05.016 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:47:05.016 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:47:05.016 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:47:05.016 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:47:05.016 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:47:05.016 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:47:05.016 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:47:05.016 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:47:05.017 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:47:05.017 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:47:05.017 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:47:05.017 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:47:05.017 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:47:05.017 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:47:05.017 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:47:05.017 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:47:05.017 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:47:05.017 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:47:05.017 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:47:05.017 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:47:05.017 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:47:05.017 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:47:05.033 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:47:05.033 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:47:05.033 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:47:05.033 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_5 2024-09-12T06:47:05.131 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:47:05.131 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror image_5 2024-09-12T06:47:05.131 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:47:05.131 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:47:05.131 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_5 2024-09-12T06:47:05.131 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_5 '' 2024-09-12T06:47:05.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:47:05.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:47:05.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_5 2024-09-12T06:47:05.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:47:05.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_5.status 2024-09-12T06:47:05.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_5.status 2024-09-12T06:47:05.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:47:05.133 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_5.status 2024-09-12T06:47:05.133 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_5 2024-09-12T06:47:05.133 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_5.status 2024-09-12T06:47:05.178 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 3 2024-09-12T06:47:05.178 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 3 2024-09-12T06:47:05.178 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:47:05.178 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=11983], [object_number=14, tag_tid=1, entry_tid=11982]]], state=connected] 2024-09-12T06:47:05.178 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=11983], [object_number=14, tag_tid=1, entry_tid=11982]]], state=connected] 2024-09-12T06:47:05.180 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_5.status 2024-09-12T06:47:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-09-12T06:47:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror image_5 2024-09-12T06:47:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:47:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:47:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_5 2024-09-12T06:47:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_5 '..*' 2024-09-12T06:47:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:47:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:47:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_5 2024-09-12T06:47:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:47:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_5.status 2024-09-12T06:47:05.184 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_5.status 2024-09-12T06:47:05.184 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:47:05.184 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_5.status 2024-09-12T06:47:05.184 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_5 2024-09-12T06:47:05.184 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_5.status 2024-09-12T06:47:05.229 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 3 2024-09-12T06:47:05.229 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 3 2024-09-12T06:47:05.229 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:47:05.229 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=11983], [object_number=14, tag_tid=1, entry_tid=11982]]], state=connected] 2024-09-12T06:47:05.229 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=11983], [object_number=14, tag_tid=1, entry_tid=11982]]], state=connected] 2024-09-12T06:47:05.232 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_5.status 2024-09-12T06:47:05.234 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-09-12T06:47:05.234 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=1' 2024-09-12T06:47:05.234 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:47:05.234 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_snap_present cluster1 mirror image_5 snap 2024-09-12T06:47:05.234 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:47:05.235 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:47:05.235 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_5 2024-09-12T06:47:05.235 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:47:05.235 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:47:05.235 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T06:47:05.235 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:47:06.235 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 info mirror/image_5@snap 2024-09-12T06:47:06.280 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:rbd image 'image_5': 2024-09-12T06:47:06.280 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: size 128 MiB in 32 objects 2024-09-12T06:47:06.280 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: order 22 (4 MiB objects) 2024-09-12T06:47:06.280 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: snapshot_count: 1 2024-09-12T06:47:06.280 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: id: 10a8c86d579c 2024-09-12T06:47:06.280 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: block_name_prefix: rbd_data.10a8c86d579c 2024-09-12T06:47:06.281 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: format: 2 2024-09-12T06:47:06.281 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: features: layering, exclusive-lock, journaling 2024-09-12T06:47:06.281 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: op_features: 2024-09-12T06:47:06.281 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: flags: 2024-09-12T06:47:06.281 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: create_timestamp: Thu Sep 12 06:39:07 2024 2024-09-12T06:47:06.281 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: access_timestamp: Thu Sep 12 06:39:07 2024 2024-09-12T06:47:06.281 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: modify_timestamp: Thu Sep 12 06:40:08 2024 2024-09-12T06:47:06.281 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: protected: False 2024-09-12T06:47:06.281 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: journal: 10a8c86d579c 2024-09-12T06:47:06.281 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring state: enabled 2024-09-12T06:47:06.281 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring mode: journal 2024-09-12T06:47:06.281 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring global id: b190281d-d9ac-4e52-abdb-52d1f596e4c6 2024-09-12T06:47:06.281 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring primary: false 2024-09-12T06:47:06.283 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:47:06.283 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror image_5 snap 2024-09-12T06:47:06.284 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:47:06.284 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_5 2024-09-12T06:47:06.284 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:47:06.284 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:47:06.284 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_5.export 2024-09-12T06:47:06.284 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_5.export 2024-09-12T06:47:06.284 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_5.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_5.export 2024-09-12T06:47:06.284 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export image_5@snap /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_5.export 2024-09-12T06:47:06.448 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:47:06.456 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export image_5@snap /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_5.export 2024-09-12T06:47:06.686 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:47:06.695 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_5.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_5.export 2024-09-12T06:47:06.751 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_5.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_5.export 2024-09-12T06:47:06.784 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:47:06.784 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:47:06.784 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_6 2024-09-12T06:47:06.784 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_snap cluster2 mirror image_6 snap 2024-09-12T06:47:06.784 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:47:06.784 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:47:06.784 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_6 2024-09-12T06:47:06.784 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:47:06.784 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror snap create image_6@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T06:47:09.675 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_started cluster1 mirror image_6 2024-09-12T06:47:09.675 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:47:09.675 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:47:09.675 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_6 2024-09-12T06:47:09.675 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_state cluster1 mirror image_6 started 2024-09-12T06:47:09.675 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:47:09.675 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:47:09.675 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_6 2024-09-12T06:47:09.675 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=started 2024-09-12T06:47:09.676 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:47:09.676 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:47:09.676 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:47:10.676 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_replay_state cluster1 mirror image_6 started 2024-09-12T06:47:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:47:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:47:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_6 2024-09-12T06:47:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=started 2024-09-12T06:47:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local status_result 2024-09-12T06:47:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=stopped 2024-09-12T06:47:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_6 2024-09-12T06:47:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -i state 2024-09-12T06:47:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_instance=cluster1 2024-09-12T06:47:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:47:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:47:10.678 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance=cluster1 2024-09-12T06:47:10.678 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local loop_instance 2024-09-12T06:47:10.678 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:47:10.678 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sleep 0 2024-09-12T06:47:10.680 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:47:10.680 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ seq 0 1 2024-09-12T06:47:10.680 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:47:10.680 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_6 2024-09-12T06:47:10.680 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:0 2024-09-12T06:47:10.680 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:47:10.680 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:47:10.680 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:47:10.680 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:47:10.681 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:47:10.681 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:47:10.681 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:47:10.681 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:47:10.681 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:47:10.681 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:47:10.681 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:47:10.681 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:47:10.681 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:47:10.681 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:47:10.681 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:0 2024-09-12T06:47:10.681 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:47:10.681 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:47:10.683 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:47:10.683 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:0 2024-09-12T06:47:10.683 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:47:10.683 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:47:10.683 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:47:10.683 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:47:10.683 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T06:47:10.683 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:47:10.683 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:47:10.683 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=0 2024-09-12T06:47:10.683 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=0 2024-09-12T06:47:10.684 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:47:10.699 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:47:10.699 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:47:10.699 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:47:10.699 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_6 2024-09-12T06:47:10.794 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:Invalid command: unused arguments: ['mirror/image_6'] 2024-09-12T06:47:10.794 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T06:47:10.794 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:admin_socket: invalid command 2024-09-12T06:47:10.798 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:47:10.798 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_6 2024-09-12T06:47:10.799 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:1 2024-09-12T06:47:10.799 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:47:10.799 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:47:10.799 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T06:47:10.799 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:1 2024-09-12T06:47:10.799 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:47:10.799 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:47:10.799 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:47:10.799 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:47:10.799 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T06:47:10.799 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:47:10.800 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:47:10.800 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=1 2024-09-12T06:47:10.800 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:47:10.800 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T06:47:10.800 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:1 2024-09-12T06:47:10.800 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:47:10.800 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:47:10.800 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T06:47:10.800 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:1 2024-09-12T06:47:10.800 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:47:10.800 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:47:10.800 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:47:10.800 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:47:10.801 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T06:47:10.801 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:47:10.801 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:47:10.801 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=1 2024-09-12T06:47:10.801 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=1 2024-09-12T06:47:10.801 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T06:47:10.816 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:47:10.816 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:47:10.816 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:47:10.816 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_6 2024-09-12T06:47:10.916 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ return 0 2024-09-12T06:47:10.916 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T06:47:10.917 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo ' "state": "Replaying"' 2024-09-12T06:47:10.917 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep -i Replaying 2024-09-12T06:47:10.918 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: "state": "Replaying" 2024-09-12T06:47:10.919 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_state=started 2024-09-12T06:47:10.919 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test started = started 2024-09-12T06:47:10.919 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:47:10.919 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_6 2024-09-12T06:47:10.919 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:47:10.919 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:47:10.920 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:47:10.920 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:47:10.920 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_6 2024-09-12T06:47:10.920 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' journal = journal ']' 2024-09-12T06:47:10.920 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror image_6 2024-09-12T06:47:10.920 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:47:10.920 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:47:10.920 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:47:10.920 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:47:10.920 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_6 2024-09-12T06:47:10.920 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-12T06:47:10.920 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-12T06:47:10.920 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:47:10.920 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:47:10.921 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:47:11.120 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror image_6 2024-09-12T06:47:11.120 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:47:11.120 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:47:11.120 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_6 2024-09-12T06:47:11.120 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:47:11.121 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n image_6 ']' 2024-09-12T06:47:11.121 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/image_6' 2024-09-12T06:47:11.121 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_6 2024-09-12T06:47:11.121 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:47:11.121 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:47:11.121 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:47:11.121 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:47:11.121 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:47:11.121 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:47:11.121 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:47:11.121 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:47:11.125 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:47:11.125 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:47:11.125 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_6 2024-09-12T06:47:11.125 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:47:11.125 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:47:11.125 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:47:11.125 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:47:11.125 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:47:11.125 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:47:11.125 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:47:11.125 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:47:11.126 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:47:11.126 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:47:11.126 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:47:11.126 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:47:11.126 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:47:11.126 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:47:11.126 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:47:11.126 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:47:11.126 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:47:11.126 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:47:11.126 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:47:11.126 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:47:11.126 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:47:11.127 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:47:11.127 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:47:11.127 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:47:11.127 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:47:11.127 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:47:11.127 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:47:11.127 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:47:11.127 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:47:11.127 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:47:11.142 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:47:11.143 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:47:11.143 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:47:11.143 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_6 2024-09-12T06:47:11.228 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:Invalid command: unused arguments: ['mirror/image_6'] 2024-09-12T06:47:11.228 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd mirror flush : flush rbd mirror 2024-09-12T06:47:11.228 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:admin_socket: invalid command 2024-09-12T06:47:11.231 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:47:11.231 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_6 2024-09-12T06:47:11.231 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:1 2024-09-12T06:47:11.231 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:47:11.232 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:47:11.232 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-12T06:47:11.232 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:1 2024-09-12T06:47:11.232 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:47:11.232 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:47:11.232 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:47:11.232 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=1 2024-09-12T06:47:11.232 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-12T06:47:11.232 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:47:11.232 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:47:11.232 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=1 2024-09-12T06:47:11.232 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:47:11.232 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-12T06:47:11.232 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:1 2024-09-12T06:47:11.233 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:47:11.233 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:47:11.233 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T06:47:11.233 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:1 2024-09-12T06:47:11.233 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:47:11.233 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:47:11.233 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:47:11.233 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:47:11.233 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T06:47:11.233 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:47:11.233 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:47:11.233 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=1 2024-09-12T06:47:11.233 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:47:11.233 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T06:47:11.250 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:47:11.250 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:47:11.250 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:47:11.250 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_6 2024-09-12T06:47:11.347 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:47:11.347 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror image_6 2024-09-12T06:47:11.347 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:47:11.347 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:47:11.347 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_6 2024-09-12T06:47:11.347 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_6 '' 2024-09-12T06:47:11.347 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:47:11.347 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:47:11.347 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_6 2024-09-12T06:47:11.347 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:47:11.348 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_6.status 2024-09-12T06:47:11.348 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_6.status 2024-09-12T06:47:11.348 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:47:11.348 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_6.status 2024-09-12T06:47:11.348 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_6 2024-09-12T06:47:11.348 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_6.status 2024-09-12T06:47:11.594 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 4 2024-09-12T06:47:11.594 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 4 2024-09-12T06:47:11.594 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:47:11.594 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=14431], [object_number=18, tag_tid=1, entry_tid=14430]]], state=connected] 2024-09-12T06:47:11.595 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=14431], [object_number=18, tag_tid=1, entry_tid=14430]]], state=connected] 2024-09-12T06:47:11.597 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_6.status 2024-09-12T06:47:11.598 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=17, tag_tid=2, entry_tid=1' 2024-09-12T06:47:11.600 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror image_6 2024-09-12T06:47:11.600 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:47:11.600 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:47:11.600 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_6 2024-09-12T06:47:11.600 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_6 '..*' 2024-09-12T06:47:11.600 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:47:11.600 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:47:11.600 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_6 2024-09-12T06:47:11.600 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:47:11.601 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_6.status 2024-09-12T06:47:11.601 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_6.status 2024-09-12T06:47:11.601 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:47:11.602 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_6.status 2024-09-12T06:47:11.602 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_6 2024-09-12T06:47:11.602 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_6.status 2024-09-12T06:47:11.646 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 4 2024-09-12T06:47:11.646 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 4 2024-09-12T06:47:11.646 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:47:11.646 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=14431], [object_number=18, tag_tid=1, entry_tid=14430]]], state=connected] 2024-09-12T06:47:11.646 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=14431], [object_number=18, tag_tid=1, entry_tid=14430]]], state=connected] 2024-09-12T06:47:11.649 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_6.status 2024-09-12T06:47:11.651 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=17, tag_tid=2, entry_tid=1' 2024-09-12T06:47:11.652 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=17, tag_tid=2, entry_tid=1' -a 'object_number=17, tag_tid=2, entry_tid=1' = 'object_number=17, tag_tid=2, entry_tid=1' 2024-09-12T06:47:11.652 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:47:11.652 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_snap_present cluster1 mirror image_6 snap 2024-09-12T06:47:11.652 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:47:11.652 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:47:11.652 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_6 2024-09-12T06:47:11.652 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:47:11.652 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:47:11.652 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T06:47:11.652 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:47:12.652 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 info mirror/image_6@snap 2024-09-12T06:47:12.699 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:rbd image 'image_6': 2024-09-12T06:47:12.699 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: size 128 MiB in 32 objects 2024-09-12T06:47:12.699 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: order 22 (4 MiB objects) 2024-09-12T06:47:12.699 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: snapshot_count: 1 2024-09-12T06:47:12.699 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: id: 10afb6c688a7 2024-09-12T06:47:12.699 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: block_name_prefix: rbd_data.10afb6c688a7 2024-09-12T06:47:12.699 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: format: 2 2024-09-12T06:47:12.699 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: features: layering, exclusive-lock, journaling 2024-09-12T06:47:12.699 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: op_features: 2024-09-12T06:47:12.699 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: flags: 2024-09-12T06:47:12.699 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: create_timestamp: Thu Sep 12 06:39:19 2024 2024-09-12T06:47:12.700 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: access_timestamp: Thu Sep 12 06:39:19 2024 2024-09-12T06:47:12.700 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: modify_timestamp: Thu Sep 12 06:40:20 2024 2024-09-12T06:47:12.700 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: protected: False 2024-09-12T06:47:12.700 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: journal: 10afb6c688a7 2024-09-12T06:47:12.700 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring state: enabled 2024-09-12T06:47:12.700 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring mode: journal 2024-09-12T06:47:12.700 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring global id: b8c72ad7-8dcd-49cb-bf41-7339e94fda39 2024-09-12T06:47:12.700 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring primary: false 2024-09-12T06:47:12.702 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:47:12.747 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror image_6 snap 2024-09-12T06:47:12.748 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:47:12.748 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_6 2024-09-12T06:47:12.748 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:47:12.748 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:47:12.748 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_6.export 2024-09-12T06:47:12.748 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_6.export 2024-09-12T06:47:12.749 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_6.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_6.export 2024-09-12T06:47:12.750 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export image_6@snap /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_6.export 2024-09-12T06:47:12.863 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:47:12.869 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export image_6@snap /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_6.export 2024-09-12T06:47:13.085 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:47:13.091 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_6.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_6.export 2024-09-12T06:47:13.146 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_6.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_6.export 2024-09-12T06:47:13.178 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:47:13.178 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:47:13.178 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_7 2024-09-12T06:47:13.178 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_snap cluster2 mirror image_7 snap 2024-09-12T06:47:13.179 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:47:13.179 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:47:13.179 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_7 2024-09-12T06:47:13.179 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:47:13.179 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror snap create image_7@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T06:47:13.686 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_started cluster1 mirror image_7 2024-09-12T06:47:13.686 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:47:13.686 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:47:13.687 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_7 2024-09-12T06:47:13.687 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_state cluster1 mirror image_7 started 2024-09-12T06:47:13.687 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:47:13.687 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:47:13.687 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_7 2024-09-12T06:47:13.687 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=started 2024-09-12T06:47:13.687 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:47:13.687 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:47:13.687 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:47:14.689 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_replay_state cluster1 mirror image_7 started 2024-09-12T06:47:14.689 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:47:14.689 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:47:14.689 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_7 2024-09-12T06:47:14.690 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=started 2024-09-12T06:47:14.690 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local status_result 2024-09-12T06:47:14.690 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=stopped 2024-09-12T06:47:14.690 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_7 2024-09-12T06:47:14.690 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -i state 2024-09-12T06:47:14.690 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_instance=cluster1 2024-09-12T06:47:14.690 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:47:14.690 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:47:14.690 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance=cluster1 2024-09-12T06:47:14.690 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local loop_instance 2024-09-12T06:47:14.690 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:47:14.691 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sleep 0 2024-09-12T06:47:14.691 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:47:14.691 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ seq 0 1 2024-09-12T06:47:14.694 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:47:14.694 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_7 2024-09-12T06:47:14.695 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:0 2024-09-12T06:47:14.695 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:47:14.695 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:47:14.695 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:47:14.695 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:47:14.695 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:47:14.695 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:47:14.695 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:47:14.695 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:47:14.695 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:47:14.695 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:47:14.695 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:47:14.696 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:47:14.696 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:47:14.696 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:47:14.696 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:0 2024-09-12T06:47:14.696 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:47:14.696 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:47:14.696 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:47:14.696 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:0 2024-09-12T06:47:14.696 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:47:14.696 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:47:14.697 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:47:14.697 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:47:14.697 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T06:47:14.697 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:47:14.697 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:47:14.697 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=0 2024-09-12T06:47:14.697 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=0 2024-09-12T06:47:14.697 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:47:14.710 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:47:14.710 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:47:14.711 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:47:14.711 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_7 2024-09-12T06:47:14.836 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ return 0 2024-09-12T06:47:14.836 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T06:47:14.836 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo ' "state": "Replaying"' 2024-09-12T06:47:14.836 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep -i Replaying 2024-09-12T06:47:14.837 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: "state": "Replaying" 2024-09-12T06:47:14.840 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_state=started 2024-09-12T06:47:14.840 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test started = started 2024-09-12T06:47:14.840 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:47:14.840 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_7 2024-09-12T06:47:14.840 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:47:14.840 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:47:14.840 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:47:14.840 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:47:14.841 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_7 2024-09-12T06:47:14.841 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' journal = journal ']' 2024-09-12T06:47:14.841 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror image_7 2024-09-12T06:47:14.841 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:47:14.841 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:47:14.841 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:47:14.841 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:47:14.841 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_7 2024-09-12T06:47:14.841 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-12T06:47:14.841 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-12T06:47:14.841 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:47:14.841 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:47:14.842 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:47:15.045 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror image_7 2024-09-12T06:47:15.046 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:47:15.046 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:47:15.046 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_7 2024-09-12T06:47:15.046 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:47:15.046 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n image_7 ']' 2024-09-12T06:47:15.046 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/image_7' 2024-09-12T06:47:15.046 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_7 2024-09-12T06:47:15.046 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:47:15.046 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:47:15.046 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:47:15.046 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:47:15.047 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:47:15.047 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:47:15.047 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:47:15.047 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:47:15.047 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:47:15.047 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:47:15.047 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_7 2024-09-12T06:47:15.047 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:47:15.047 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:47:15.047 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:47:15.047 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:47:15.047 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:47:15.047 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:47:15.047 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:47:15.048 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:47:15.048 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:47:15.048 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:47:15.048 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:47:15.048 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:47:15.048 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:47:15.048 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:47:15.048 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:47:15.048 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:47:15.048 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:47:15.048 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:47:15.048 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:47:15.049 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:47:15.049 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:47:15.049 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:47:15.049 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:47:15.049 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:47:15.049 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:47:15.049 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:47:15.049 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:47:15.049 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:47:15.049 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:47:15.049 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:47:15.060 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:47:15.061 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:47:15.061 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:47:15.061 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_7 2024-09-12T06:47:15.160 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:47:15.160 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror image_7 2024-09-12T06:47:15.160 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:47:15.160 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:47:15.160 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_7 2024-09-12T06:47:15.160 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_7 '' 2024-09-12T06:47:15.160 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:47:15.160 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:47:15.161 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_7 2024-09-12T06:47:15.161 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:47:15.161 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_7.status 2024-09-12T06:47:15.161 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_7.status 2024-09-12T06:47:15.161 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:47:15.161 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_7.status 2024-09-12T06:47:15.162 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_7 2024-09-12T06:47:15.162 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_7.status 2024-09-12T06:47:15.206 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 1 2024-09-12T06:47:15.206 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 1 2024-09-12T06:47:15.206 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:47:15.207 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6387], [object_number=6, tag_tid=1, entry_tid=6386]]], state=connected] 2024-09-12T06:47:15.207 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6387], [object_number=6, tag_tid=1, entry_tid=6386]]], state=connected] 2024-09-12T06:47:15.208 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_7.status 2024-09-12T06:47:15.210 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-12T06:47:15.211 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror image_7 2024-09-12T06:47:15.211 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:47:15.211 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:47:15.211 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_7 2024-09-12T06:47:15.211 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_7 '..*' 2024-09-12T06:47:15.211 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:47:15.212 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:47:15.212 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_7 2024-09-12T06:47:15.212 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:47:15.212 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_7.status 2024-09-12T06:47:15.212 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_7.status 2024-09-12T06:47:15.212 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:47:15.212 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_7.status 2024-09-12T06:47:15.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_7 2024-09-12T06:47:15.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_7.status 2024-09-12T06:47:15.257 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 1 2024-09-12T06:47:15.258 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 1 2024-09-12T06:47:15.258 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:47:15.258 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6387], [object_number=6, tag_tid=1, entry_tid=6386]]], state=connected] 2024-09-12T06:47:15.258 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6387], [object_number=6, tag_tid=1, entry_tid=6386]]], state=connected] 2024-09-12T06:47:15.260 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_7.status 2024-09-12T06:47:15.262 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-12T06:47:15.263 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-09-12T06:47:15.263 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:47:15.263 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_snap_present cluster1 mirror image_7 snap 2024-09-12T06:47:15.263 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:47:15.263 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:47:15.263 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_7 2024-09-12T06:47:15.263 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:47:15.263 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:47:15.263 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T06:47:15.263 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:47:16.263 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 info mirror/image_7@snap 2024-09-12T06:47:16.311 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:rbd image 'image_7': 2024-09-12T06:47:16.311 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: size 128 MiB in 32 objects 2024-09-12T06:47:16.311 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: order 22 (4 MiB objects) 2024-09-12T06:47:16.311 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: snapshot_count: 1 2024-09-12T06:47:16.311 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: id: 10a85eece893 2024-09-12T06:47:16.311 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: block_name_prefix: rbd_data.10a85eece893 2024-09-12T06:47:16.311 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: format: 2 2024-09-12T06:47:16.311 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: features: layering, exclusive-lock, journaling 2024-09-12T06:47:16.311 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: op_features: 2024-09-12T06:47:16.312 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: flags: 2024-09-12T06:47:16.312 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: create_timestamp: Thu Sep 12 06:39:33 2024 2024-09-12T06:47:16.312 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: access_timestamp: Thu Sep 12 06:39:33 2024 2024-09-12T06:47:16.312 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: modify_timestamp: Thu Sep 12 06:40:34 2024 2024-09-12T06:47:16.312 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: protected: False 2024-09-12T06:47:16.312 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: journal: 10a85eece893 2024-09-12T06:47:16.312 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring state: enabled 2024-09-12T06:47:16.312 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring mode: journal 2024-09-12T06:47:16.312 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring global id: 41d4f35e-a102-45ac-8ec5-256b6b082f69 2024-09-12T06:47:16.313 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring primary: false 2024-09-12T06:47:16.314 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:47:16.314 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror image_7 snap 2024-09-12T06:47:16.315 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:47:16.315 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_7 2024-09-12T06:47:16.315 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:47:16.315 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:47:16.315 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_7.export 2024-09-12T06:47:16.315 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_7.export 2024-09-12T06:47:16.315 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_7.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_7.export 2024-09-12T06:47:16.315 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export image_7@snap /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_7.export 2024-09-12T06:47:16.473 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:47:16.481 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export image_7@snap /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_7.export 2024-09-12T06:47:16.661 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:47:16.668 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_7.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_7.export 2024-09-12T06:47:16.723 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_7.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_7.export 2024-09-12T06:47:16.755 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:47:16.755 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:47:16.755 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_8 2024-09-12T06:47:16.756 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_snap cluster2 mirror image_8 snap 2024-09-12T06:47:16.756 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:47:16.756 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:47:16.756 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_8 2024-09-12T06:47:16.756 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:47:16.756 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror snap create image_8@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T06:47:18.697 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_started cluster1 mirror image_8 2024-09-12T06:47:18.697 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:47:18.697 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:47:18.698 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_8 2024-09-12T06:47:18.698 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_state cluster1 mirror image_8 started 2024-09-12T06:47:18.698 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:47:18.698 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:47:18.698 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_8 2024-09-12T06:47:18.698 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=started 2024-09-12T06:47:18.698 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:47:18.698 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:47:18.698 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:47:19.698 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_replay_state cluster1 mirror image_8 started 2024-09-12T06:47:19.698 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:47:19.698 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:47:19.698 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_8 2024-09-12T06:47:19.698 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=started 2024-09-12T06:47:19.698 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local status_result 2024-09-12T06:47:19.699 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=stopped 2024-09-12T06:47:19.700 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_8 2024-09-12T06:47:19.700 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_instance=cluster1 2024-09-12T06:47:19.700 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:47:19.700 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -i state 2024-09-12T06:47:19.700 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:47:19.700 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance=cluster1 2024-09-12T06:47:19.700 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local loop_instance 2024-09-12T06:47:19.700 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:47:19.700 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sleep 0 2024-09-12T06:47:19.701 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:47:19.701 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ seq 0 1 2024-09-12T06:47:19.703 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:47:19.704 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_8 2024-09-12T06:47:19.704 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:0 2024-09-12T06:47:19.704 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:47:19.704 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:47:19.704 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:47:19.704 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:47:19.704 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:47:19.704 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:47:19.704 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:47:19.704 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:47:19.704 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:47:19.704 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:47:19.705 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:47:19.705 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:47:19.705 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:47:19.705 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:47:19.705 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:0 2024-09-12T06:47:19.705 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:47:19.705 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:47:19.705 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:47:19.705 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:0 2024-09-12T06:47:19.706 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:47:19.706 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:47:19.706 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:47:19.706 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:47:19.706 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T06:47:19.706 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:47:19.706 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:47:19.706 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=0 2024-09-12T06:47:19.706 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=0 2024-09-12T06:47:19.706 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:47:19.726 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:47:19.726 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:47:19.726 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:47:19.726 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_8 2024-09-12T06:47:19.848 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2024-09-12T06:47:19.849 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T06:47:19.849 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:admin_socket: invalid command 2024-09-12T06:47:19.852 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:47:19.853 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_8 2024-09-12T06:47:19.853 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:1 2024-09-12T06:47:19.853 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:47:19.853 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:47:19.853 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T06:47:19.853 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:1 2024-09-12T06:47:19.853 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:47:19.853 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:47:19.853 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:47:19.853 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:47:19.853 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T06:47:19.853 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:47:19.854 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:47:19.854 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=1 2024-09-12T06:47:19.854 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:47:19.854 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T06:47:19.854 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:1 2024-09-12T06:47:19.854 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:47:19.854 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:47:19.854 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T06:47:19.854 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:1 2024-09-12T06:47:19.854 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:47:19.854 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:47:19.854 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:47:19.854 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:47:19.854 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T06:47:19.855 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:47:19.855 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:47:19.855 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=1 2024-09-12T06:47:19.855 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=1 2024-09-12T06:47:19.855 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T06:47:19.871 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:47:19.871 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:47:19.872 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:47:19.872 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_8 2024-09-12T06:47:19.979 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ return 0 2024-09-12T06:47:19.979 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T06:47:19.979 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo ' "state": "Replaying"' 2024-09-12T06:47:19.980 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep -i Replaying 2024-09-12T06:47:19.981 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: "state": "Replaying" 2024-09-12T06:47:19.983 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_state=started 2024-09-12T06:47:19.983 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test started = started 2024-09-12T06:47:19.983 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:47:19.983 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_8 2024-09-12T06:47:19.983 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:47:19.983 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:47:19.983 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:47:19.983 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:47:19.983 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_8 2024-09-12T06:47:19.983 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' journal = journal ']' 2024-09-12T06:47:19.983 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror image_8 2024-09-12T06:47:19.983 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:47:19.984 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:47:19.984 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:47:19.984 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:47:19.984 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_8 2024-09-12T06:47:19.984 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-12T06:47:19.984 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-12T06:47:19.984 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:47:19.984 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:47:19.984 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:47:20.183 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror image_8 2024-09-12T06:47:20.184 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:47:20.184 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:47:20.184 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_8 2024-09-12T06:47:20.184 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:47:20.184 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n image_8 ']' 2024-09-12T06:47:20.184 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/image_8' 2024-09-12T06:47:20.184 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_8 2024-09-12T06:47:20.184 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:47:20.185 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:47:20.185 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:47:20.185 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:47:20.185 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:47:20.185 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:47:20.185 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:47:20.185 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:47:20.185 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:47:20.188 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:47:20.188 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_8 2024-09-12T06:47:20.189 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:47:20.189 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:47:20.189 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:47:20.189 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:47:20.189 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:47:20.189 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:47:20.189 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:47:20.189 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:47:20.189 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:47:20.189 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:47:20.189 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:47:20.189 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:47:20.189 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:47:20.190 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:47:20.190 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:47:20.190 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:47:20.190 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:47:20.190 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:47:20.190 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:47:20.190 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:47:20.190 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:47:20.191 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:47:20.191 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:47:20.191 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:47:20.191 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:47:20.191 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:47:20.191 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:47:20.191 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:47:20.191 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:47:20.191 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:47:20.205 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:47:20.205 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:47:20.205 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:47:20.205 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_8 2024-09-12T06:47:20.291 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2024-09-12T06:47:20.291 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd mirror flush : flush rbd mirror 2024-09-12T06:47:20.291 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:admin_socket: invalid command 2024-09-12T06:47:20.294 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:47:20.295 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_8 2024-09-12T06:47:20.295 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:1 2024-09-12T06:47:20.295 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:47:20.295 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:47:20.295 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-12T06:47:20.295 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:1 2024-09-12T06:47:20.295 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:47:20.295 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:47:20.295 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:47:20.295 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=1 2024-09-12T06:47:20.295 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-12T06:47:20.295 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:47:20.295 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:47:20.296 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=1 2024-09-12T06:47:20.296 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:47:20.296 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-12T06:47:20.296 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:1 2024-09-12T06:47:20.296 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:47:20.296 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:47:20.296 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T06:47:20.296 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:1 2024-09-12T06:47:20.296 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:47:20.296 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:47:20.296 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:47:20.296 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:47:20.296 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T06:47:20.296 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:47:20.297 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:47:20.297 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=1 2024-09-12T06:47:20.297 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:47:20.297 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T06:47:20.313 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:47:20.313 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:47:20.313 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:47:20.313 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_8 2024-09-12T06:47:20.412 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:47:20.412 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror image_8 2024-09-12T06:47:20.412 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:47:20.412 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:47:20.413 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_8 2024-09-12T06:47:20.413 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_8 '' 2024-09-12T06:47:20.413 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:47:20.413 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:47:20.413 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_8 2024-09-12T06:47:20.413 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:47:20.413 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_8.status 2024-09-12T06:47:20.413 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_8.status 2024-09-12T06:47:20.413 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:47:20.414 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_8.status 2024-09-12T06:47:20.414 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_8 2024-09-12T06:47:20.414 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_8.status 2024-09-12T06:47:20.461 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 4 2024-09-12T06:47:20.462 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 4 2024-09-12T06:47:20.462 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:47:20.462 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=18, tag_tid=1, entry_tid=14174], [object_number=19, tag_tid=1, entry_tid=14171]]], state=connected] 2024-09-12T06:47:20.462 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=18, tag_tid=1, entry_tid=14174], [object_number=19, tag_tid=1, entry_tid=14171]]], state=connected] 2024-09-12T06:47:20.465 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_8.status 2024-09-12T06:47:20.466 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=17, tag_tid=2, entry_tid=1' 2024-09-12T06:47:20.467 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror image_8 2024-09-12T06:47:20.468 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:47:20.468 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:47:20.468 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_8 2024-09-12T06:47:20.468 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_8 '..*' 2024-09-12T06:47:20.468 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:47:20.468 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:47:20.468 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_8 2024-09-12T06:47:20.468 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:47:20.468 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_8.status 2024-09-12T06:47:20.468 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_8.status 2024-09-12T06:47:20.468 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:47:20.469 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_8.status 2024-09-12T06:47:20.469 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_8 2024-09-12T06:47:20.469 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_8.status 2024-09-12T06:47:20.515 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 4 2024-09-12T06:47:20.515 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 4 2024-09-12T06:47:20.515 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:47:20.515 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=18, tag_tid=1, entry_tid=14174], [object_number=19, tag_tid=1, entry_tid=14171]]], state=connected] 2024-09-12T06:47:20.515 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=18, tag_tid=1, entry_tid=14174], [object_number=19, tag_tid=1, entry_tid=14171]]], state=connected] 2024-09-12T06:47:20.518 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_8.status 2024-09-12T06:47:20.520 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=17, tag_tid=2, entry_tid=1' 2024-09-12T06:47:20.520 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=17, tag_tid=2, entry_tid=1' -a 'object_number=17, tag_tid=2, entry_tid=1' = 'object_number=17, tag_tid=2, entry_tid=1' 2024-09-12T06:47:20.520 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:47:20.521 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_snap_present cluster1 mirror image_8 snap 2024-09-12T06:47:20.521 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:47:20.521 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:47:20.521 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_8 2024-09-12T06:47:20.521 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:47:20.521 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:47:20.521 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T06:47:20.521 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:47:21.521 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 info mirror/image_8@snap 2024-09-12T06:47:21.567 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:rbd image 'image_8': 2024-09-12T06:47:21.567 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: size 128 MiB in 32 objects 2024-09-12T06:47:21.567 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: order 22 (4 MiB objects) 2024-09-12T06:47:21.568 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: snapshot_count: 1 2024-09-12T06:47:21.568 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: id: 10af55c4a99c 2024-09-12T06:47:21.568 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: block_name_prefix: rbd_data.10af55c4a99c 2024-09-12T06:47:21.568 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: format: 2 2024-09-12T06:47:21.568 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: features: layering, exclusive-lock, journaling 2024-09-12T06:47:21.568 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: op_features: 2024-09-12T06:47:21.568 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: flags: 2024-09-12T06:47:21.568 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: create_timestamp: Thu Sep 12 06:39:40 2024 2024-09-12T06:47:21.569 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: access_timestamp: Thu Sep 12 06:39:40 2024 2024-09-12T06:47:21.569 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: modify_timestamp: Thu Sep 12 06:40:41 2024 2024-09-12T06:47:21.569 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: protected: False 2024-09-12T06:47:21.569 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: journal: 10af55c4a99c 2024-09-12T06:47:21.569 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring state: enabled 2024-09-12T06:47:21.569 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring mode: journal 2024-09-12T06:47:21.569 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring global id: 28475c2a-4581-4813-aa44-02757b3627eb 2024-09-12T06:47:21.569 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring primary: false 2024-09-12T06:47:21.571 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:47:21.571 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror image_8 snap 2024-09-12T06:47:21.571 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:47:21.571 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_8 2024-09-12T06:47:21.572 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:47:21.572 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:47:21.572 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_8.export 2024-09-12T06:47:21.572 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_8.export 2024-09-12T06:47:21.572 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_8.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_8.export 2024-09-12T06:47:21.573 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export image_8@snap /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_8.export 2024-09-12T06:47:21.717 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:47:21.724 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export image_8@snap /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_8.export 2024-09-12T06:47:21.918 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:47:21.924 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_8.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_8.export 2024-09-12T06:47:21.981 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_8.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_8.export 2024-09-12T06:47:22.013 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:47:22.013 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:47:22.013 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_9 2024-09-12T06:47:22.013 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_snap cluster2 mirror image_9 snap 2024-09-12T06:47:22.013 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:47:22.013 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:47:22.013 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_9 2024-09-12T06:47:22.013 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:47:22.013 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror snap create image_9@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T06:47:22.711 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_started cluster1 mirror image_9 2024-09-12T06:47:22.712 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:47:22.712 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:47:22.712 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_9 2024-09-12T06:47:22.712 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_state cluster1 mirror image_9 started 2024-09-12T06:47:22.712 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:47:22.712 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:47:22.712 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_9 2024-09-12T06:47:22.712 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=started 2024-09-12T06:47:22.712 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:47:22.712 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:47:22.712 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:47:23.712 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_replay_state cluster1 mirror image_9 started 2024-09-12T06:47:23.713 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:47:23.713 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:47:23.713 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_9 2024-09-12T06:47:23.713 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=started 2024-09-12T06:47:23.713 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local status_result 2024-09-12T06:47:23.713 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=stopped 2024-09-12T06:47:23.714 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_9 2024-09-12T06:47:23.714 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_instance=cluster1 2024-09-12T06:47:23.714 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -i state 2024-09-12T06:47:23.714 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:47:23.715 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:47:23.715 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance=cluster1 2024-09-12T06:47:23.715 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local loop_instance 2024-09-12T06:47:23.715 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:47:23.715 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sleep 0 2024-09-12T06:47:23.715 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:47:23.715 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ seq 0 1 2024-09-12T06:47:23.717 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:47:23.717 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_9 2024-09-12T06:47:23.717 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:0 2024-09-12T06:47:23.717 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:47:23.717 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:47:23.718 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:47:23.718 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:47:23.718 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:47:23.718 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:47:23.718 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:47:23.718 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:47:23.718 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:47:23.718 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:47:23.718 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:47:23.718 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:47:23.718 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:47:23.718 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:47:23.718 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:0 2024-09-12T06:47:23.718 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:47:23.719 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:47:23.719 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:47:23.719 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:0 2024-09-12T06:47:23.720 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:47:23.720 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:47:23.720 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:47:23.720 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:47:23.721 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T06:47:23.721 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:47:23.721 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:47:23.721 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=0 2024-09-12T06:47:23.721 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=0 2024-09-12T06:47:23.721 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:47:23.736 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:47:23.736 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:47:23.736 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:47:23.736 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_9 2024-09-12T06:47:23.837 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ return 0 2024-09-12T06:47:23.838 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T06:47:23.838 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo ' "state": "Replaying"' 2024-09-12T06:47:23.838 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep -i Replaying 2024-09-12T06:47:23.839 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: "state": "Replaying" 2024-09-12T06:47:23.841 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_state=started 2024-09-12T06:47:23.841 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test started = started 2024-09-12T06:47:23.841 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:47:23.841 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_9 2024-09-12T06:47:23.841 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:47:23.841 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:47:23.841 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:47:23.841 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:47:23.841 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_9 2024-09-12T06:47:23.841 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' journal = journal ']' 2024-09-12T06:47:23.841 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror image_9 2024-09-12T06:47:23.841 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:47:23.842 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:47:23.842 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:47:23.842 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:47:23.842 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_9 2024-09-12T06:47:23.842 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-12T06:47:23.842 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-12T06:47:23.842 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:47:23.842 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:47:23.842 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:47:24.042 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror image_9 2024-09-12T06:47:24.042 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:47:24.042 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:47:24.042 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_9 2024-09-12T06:47:24.042 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:47:24.042 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n image_9 ']' 2024-09-12T06:47:24.042 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/image_9' 2024-09-12T06:47:24.042 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_9 2024-09-12T06:47:24.042 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:47:24.043 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:47:24.043 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:47:24.043 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:47:24.043 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:47:24.043 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:47:24.043 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:47:24.043 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:47:24.043 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:47:24.046 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:47:24.046 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_9 2024-09-12T06:47:24.046 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:47:24.046 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:47:24.046 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:47:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:47:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:47:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:47:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:47:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:47:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:47:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:47:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:47:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:47:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:47:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:47:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:47:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:47:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:47:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:47:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:47:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:47:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:47:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:47:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:47:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:47:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:47:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:47:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:47:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:47:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:47:24.049 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:47:24.063 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:47:24.063 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:47:24.063 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:47:24.063 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_9 2024-09-12T06:47:24.161 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:47:24.161 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror image_9 2024-09-12T06:47:24.161 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:47:24.161 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:47:24.161 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_9 2024-09-12T06:47:24.161 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_9 '' 2024-09-12T06:47:24.161 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:47:24.161 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:47:24.161 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_9 2024-09-12T06:47:24.162 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:47:24.162 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_9.status 2024-09-12T06:47:24.162 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_9.status 2024-09-12T06:47:24.162 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:47:24.162 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_9.status 2024-09-12T06:47:24.162 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_9 2024-09-12T06:47:24.163 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_9.status 2024-09-12T06:47:24.207 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 1 2024-09-12T06:47:24.207 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 1 2024-09-12T06:47:24.207 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:47:24.208 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4563], [object_number=6, tag_tid=1, entry_tid=4562]]], state=connected] 2024-09-12T06:47:24.208 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4563], [object_number=6, tag_tid=1, entry_tid=4562]]], state=connected] 2024-09-12T06:47:24.210 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_9.status 2024-09-12T06:47:24.211 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-12T06:47:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror image_9 2024-09-12T06:47:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:47:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:47:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_9 2024-09-12T06:47:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_9 '..*' 2024-09-12T06:47:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:47:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:47:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_9 2024-09-12T06:47:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:47:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_9.status 2024-09-12T06:47:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_9.status 2024-09-12T06:47:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:47:24.214 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_9.status 2024-09-12T06:47:24.214 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_9 2024-09-12T06:47:24.214 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_9.status 2024-09-12T06:47:24.258 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 1 2024-09-12T06:47:24.259 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 1 2024-09-12T06:47:24.259 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:47:24.259 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4563], [object_number=6, tag_tid=1, entry_tid=4562]]], state=connected] 2024-09-12T06:47:24.259 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4563], [object_number=6, tag_tid=1, entry_tid=4562]]], state=connected] 2024-09-12T06:47:24.261 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_9.status 2024-09-12T06:47:24.263 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-12T06:47:24.264 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-09-12T06:47:24.264 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:47:24.264 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_snap_present cluster1 mirror image_9 snap 2024-09-12T06:47:24.264 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:47:24.264 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:47:24.264 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_9 2024-09-12T06:47:24.264 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:47:24.264 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:47:24.264 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T06:47:24.264 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:47:25.264 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 info mirror/image_9@snap 2024-09-12T06:47:25.314 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:rbd image 'image_9': 2024-09-12T06:47:25.314 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: size 128 MiB in 32 objects 2024-09-12T06:47:25.314 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: order 22 (4 MiB objects) 2024-09-12T06:47:25.315 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: snapshot_count: 1 2024-09-12T06:47:25.315 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: id: 10a8b7337c71 2024-09-12T06:47:25.315 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: block_name_prefix: rbd_data.10a8b7337c71 2024-09-12T06:47:25.315 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: format: 2 2024-09-12T06:47:25.315 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: features: layering, exclusive-lock, journaling 2024-09-12T06:47:25.315 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: op_features: 2024-09-12T06:47:25.315 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: flags: 2024-09-12T06:47:25.315 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: create_timestamp: Thu Sep 12 06:39:53 2024 2024-09-12T06:47:25.315 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: access_timestamp: Thu Sep 12 06:39:53 2024 2024-09-12T06:47:25.315 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: modify_timestamp: Thu Sep 12 06:40:54 2024 2024-09-12T06:47:25.315 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: protected: False 2024-09-12T06:47:25.315 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: journal: 10a8b7337c71 2024-09-12T06:47:25.315 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring state: enabled 2024-09-12T06:47:25.315 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring mode: journal 2024-09-12T06:47:25.316 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring global id: 259e34a4-0cdf-4fe5-84aa-285d50ae5b5f 2024-09-12T06:47:25.316 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring primary: false 2024-09-12T06:47:25.318 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:47:25.318 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror image_9 snap 2024-09-12T06:47:25.318 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:47:25.318 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_9 2024-09-12T06:47:25.318 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:47:25.318 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:47:25.318 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_9.export 2024-09-12T06:47:25.318 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_9.export 2024-09-12T06:47:25.319 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_9.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_9.export 2024-09-12T06:47:25.319 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export image_9@snap /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_9.export 2024-09-12T06:47:25.471 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:47:25.479 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export image_9@snap /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_9.export 2024-09-12T06:47:25.649 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:47:25.655 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_9.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_9.export 2024-09-12T06:47:25.711 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_9.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_9.export 2024-09-12T06:47:25.743 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:47:25.743 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:47:25.743 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_10 2024-09-12T06:47:25.744 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_snap cluster2 mirror image_10 snap 2024-09-12T06:47:25.744 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:47:25.744 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:47:25.744 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_10 2024-09-12T06:47:25.744 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:47:25.744 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror snap create image_10@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T06:47:28.724 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_started cluster1 mirror image_10 2024-09-12T06:47:28.724 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:47:28.724 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:47:28.724 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_10 2024-09-12T06:47:28.725 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_state cluster1 mirror image_10 started 2024-09-12T06:47:28.725 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:47:28.725 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:47:28.725 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_10 2024-09-12T06:47:28.725 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=started 2024-09-12T06:47:28.725 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:47:28.725 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:47:28.725 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:47:29.725 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_replay_state cluster1 mirror image_10 started 2024-09-12T06:47:29.726 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:47:29.726 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:47:29.726 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_10 2024-09-12T06:47:29.726 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=started 2024-09-12T06:47:29.726 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local status_result 2024-09-12T06:47:29.726 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=stopped 2024-09-12T06:47:29.726 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_10 2024-09-12T06:47:29.726 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_instance=cluster1 2024-09-12T06:47:29.726 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:47:29.726 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -i state 2024-09-12T06:47:29.726 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:47:29.727 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance=cluster1 2024-09-12T06:47:29.727 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local loop_instance 2024-09-12T06:47:29.727 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:47:29.727 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sleep 0 2024-09-12T06:47:29.727 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:47:29.731 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ seq 0 1 2024-09-12T06:47:29.731 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:47:29.731 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_10 2024-09-12T06:47:29.731 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:0 2024-09-12T06:47:29.731 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:47:29.731 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:47:29.731 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:47:29.732 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:47:29.732 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:47:29.732 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:47:29.732 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:47:29.732 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:47:29.732 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:47:29.732 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:47:29.732 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:47:29.732 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:47:29.733 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:47:29.733 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:47:29.733 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:0 2024-09-12T06:47:29.733 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:47:29.733 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:47:29.733 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:47:29.733 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:0 2024-09-12T06:47:29.734 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:47:29.734 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:47:29.734 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:47:29.734 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:47:29.734 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T06:47:29.734 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:47:29.734 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:47:29.734 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=0 2024-09-12T06:47:29.734 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=0 2024-09-12T06:47:29.735 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:47:29.748 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:47:29.748 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:47:29.748 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:47:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_10 2024-09-12T06:47:29.861 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2024-09-12T06:47:29.861 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T06:47:29.861 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:admin_socket: invalid command 2024-09-12T06:47:29.865 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:47:29.865 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_10 2024-09-12T06:47:29.865 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:1 2024-09-12T06:47:29.865 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:47:29.865 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:47:29.865 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T06:47:29.865 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:1 2024-09-12T06:47:29.866 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:47:29.866 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:47:29.866 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:47:29.866 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:47:29.866 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T06:47:29.866 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:47:29.866 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:47:29.866 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=1 2024-09-12T06:47:29.866 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:47:29.866 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T06:47:29.866 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:1 2024-09-12T06:47:29.866 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:47:29.866 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:47:29.867 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T06:47:29.867 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:1 2024-09-12T06:47:29.867 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:47:29.867 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:47:29.867 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:47:29.867 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:47:29.867 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T06:47:29.867 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:47:29.867 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:47:29.868 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=1 2024-09-12T06:47:29.868 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=1 2024-09-12T06:47:29.868 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T06:47:29.884 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:47:29.884 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:47:29.884 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:47:29.885 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_10 2024-09-12T06:47:29.986 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ return 0 2024-09-12T06:47:29.987 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T06:47:29.987 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo ' "state": "Replaying"' 2024-09-12T06:47:29.987 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep -i Replaying 2024-09-12T06:47:29.987 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: "state": "Replaying" 2024-09-12T06:47:29.990 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_state=started 2024-09-12T06:47:29.990 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test started = started 2024-09-12T06:47:29.990 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:47:29.990 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_10 2024-09-12T06:47:29.990 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:47:29.990 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:47:29.990 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:47:29.990 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:47:29.991 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_10 2024-09-12T06:47:29.991 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' journal = journal ']' 2024-09-12T06:47:29.991 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror image_10 2024-09-12T06:47:29.991 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:47:29.991 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:47:29.991 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:47:29.992 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:47:29.992 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_10 2024-09-12T06:47:29.992 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-12T06:47:29.992 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-12T06:47:29.992 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:47:29.992 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:47:29.992 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:47:30.193 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror image_10 2024-09-12T06:47:30.193 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:47:30.193 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:47:30.193 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_10 2024-09-12T06:47:30.194 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:47:30.194 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n image_10 ']' 2024-09-12T06:47:30.194 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/image_10' 2024-09-12T06:47:30.194 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_10 2024-09-12T06:47:30.194 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:47:30.194 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:47:30.195 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:47:30.195 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:47:30.195 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:47:30.195 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:47:30.195 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:47:30.195 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:47:30.195 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:47:30.195 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:47:30.196 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_10 2024-09-12T06:47:30.196 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:47:30.196 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:47:30.196 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:47:30.196 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:47:30.196 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:47:30.196 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:47:30.196 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:47:30.196 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:47:30.196 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:47:30.196 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:47:30.196 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:47:30.196 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:47:30.197 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:47:30.197 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:47:30.197 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:47:30.197 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:47:30.197 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:47:30.197 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:47:30.197 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:47:30.197 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:47:30.197 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:47:30.197 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:47:30.197 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:47:30.197 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:47:30.197 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:47:30.198 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:47:30.198 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:47:30.198 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:47:30.198 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:47:30.198 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:47:30.212 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:47:30.212 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:47:30.212 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:47:30.212 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_10 2024-09-12T06:47:30.305 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2024-09-12T06:47:30.305 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd mirror flush : flush rbd mirror 2024-09-12T06:47:30.305 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:admin_socket: invalid command 2024-09-12T06:47:30.308 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:47:30.308 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_10 2024-09-12T06:47:30.309 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:1 2024-09-12T06:47:30.309 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:47:30.309 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:47:30.309 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-12T06:47:30.309 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:1 2024-09-12T06:47:30.309 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:47:30.309 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:47:30.309 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:47:30.309 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=1 2024-09-12T06:47:30.309 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-12T06:47:30.309 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:47:30.309 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:47:30.309 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=1 2024-09-12T06:47:30.310 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:47:30.310 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-12T06:47:30.310 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:1 2024-09-12T06:47:30.310 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:47:30.310 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:47:30.310 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T06:47:30.310 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:1 2024-09-12T06:47:30.310 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:47:30.310 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:47:30.310 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:47:30.310 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:47:30.310 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T06:47:30.310 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:47:30.310 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:47:30.311 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=1 2024-09-12T06:47:30.311 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:47:30.311 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T06:47:30.327 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:47:30.327 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:47:30.327 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:47:30.327 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_10 2024-09-12T06:47:30.425 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:47:30.425 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror image_10 2024-09-12T06:47:30.425 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:47:30.425 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:47:30.425 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_10 2024-09-12T06:47:30.425 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_10 '' 2024-09-12T06:47:30.425 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:47:30.426 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:47:30.426 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_10 2024-09-12T06:47:30.426 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:47:30.426 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_10.status 2024-09-12T06:47:30.426 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_10.status 2024-09-12T06:47:30.426 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:47:30.427 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_10.status 2024-09-12T06:47:30.427 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_10 2024-09-12T06:47:30.427 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_10.status 2024-09-12T06:47:30.473 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 3 2024-09-12T06:47:30.473 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 3 2024-09-12T06:47:30.473 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:47:30.473 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=11375], [object_number=14, tag_tid=1, entry_tid=11374]]], state=connected] 2024-09-12T06:47:30.473 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=11375], [object_number=14, tag_tid=1, entry_tid=11374]]], state=connected] 2024-09-12T06:47:30.475 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_10.status 2024-09-12T06:47:30.478 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-09-12T06:47:30.478 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror image_10 2024-09-12T06:47:30.478 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:47:30.478 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:47:30.478 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_10 2024-09-12T06:47:30.478 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_10 '..*' 2024-09-12T06:47:30.478 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:47:30.478 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:47:30.478 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_10 2024-09-12T06:47:30.478 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:47:30.478 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_10.status 2024-09-12T06:47:30.479 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_10.status 2024-09-12T06:47:30.479 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:47:30.479 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_10.status 2024-09-12T06:47:30.479 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_10 2024-09-12T06:47:30.479 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_10.status 2024-09-12T06:47:30.525 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 3 2024-09-12T06:47:30.525 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 3 2024-09-12T06:47:30.526 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:47:30.526 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=11375], [object_number=14, tag_tid=1, entry_tid=11374]]], state=connected] 2024-09-12T06:47:30.526 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=11375], [object_number=14, tag_tid=1, entry_tid=11374]]], state=connected] 2024-09-12T06:47:30.601 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_10.status 2024-09-12T06:47:30.603 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-09-12T06:47:30.603 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=1' 2024-09-12T06:47:30.603 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:47:30.603 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_snap_present cluster1 mirror image_10 snap 2024-09-12T06:47:30.603 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:47:30.603 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:47:30.603 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_10 2024-09-12T06:47:30.603 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:47:30.603 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:47:30.603 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T06:47:30.604 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:47:31.604 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 info mirror/image_10@snap 2024-09-12T06:47:31.652 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:rbd image 'image_10': 2024-09-12T06:47:31.652 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: size 128 MiB in 32 objects 2024-09-12T06:47:31.652 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: order 22 (4 MiB objects) 2024-09-12T06:47:31.652 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: snapshot_count: 1 2024-09-12T06:47:31.653 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: id: 10afabd12767 2024-09-12T06:47:31.653 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: block_name_prefix: rbd_data.10afabd12767 2024-09-12T06:47:31.653 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: format: 2 2024-09-12T06:47:31.653 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: features: layering, exclusive-lock, journaling 2024-09-12T06:47:31.653 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: op_features: 2024-09-12T06:47:31.653 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: flags: 2024-09-12T06:47:31.653 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: create_timestamp: Thu Sep 12 06:39:58 2024 2024-09-12T06:47:31.653 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: access_timestamp: Thu Sep 12 06:39:58 2024 2024-09-12T06:47:31.653 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: modify_timestamp: Thu Sep 12 06:40:59 2024 2024-09-12T06:47:31.653 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: protected: False 2024-09-12T06:47:31.653 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: journal: 10afabd12767 2024-09-12T06:47:31.653 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring state: enabled 2024-09-12T06:47:31.653 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring mode: journal 2024-09-12T06:47:31.653 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring global id: 1143e9f4-16ed-493a-b76c-c43cc730eba9 2024-09-12T06:47:31.654 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring primary: false 2024-09-12T06:47:31.656 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:47:31.656 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror image_10 snap 2024-09-12T06:47:31.656 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:47:31.656 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_10 2024-09-12T06:47:31.657 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:47:31.657 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:47:31.657 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_10.export 2024-09-12T06:47:31.657 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_10.export 2024-09-12T06:47:31.657 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_10.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_10.export 2024-09-12T06:47:31.657 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export image_10@snap /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_10.export 2024-09-12T06:47:31.812 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:47:31.819 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export image_10@snap /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_10.export 2024-09-12T06:47:31.966 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:47:31.968+0000 7fa3126a7640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-12T06:47:32.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:47:32.057 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_10.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_10.export 2024-09-12T06:47:32.112 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_10.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_10.export 2024-09-12T06:47:32.145 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:47:32.145 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:47:32.145 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_11 2024-09-12T06:47:32.145 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_snap cluster2 mirror image_11 snap 2024-09-12T06:47:32.146 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:47:32.146 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:47:32.146 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_11 2024-09-12T06:47:32.146 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:47:32.146 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror snap create image_11@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T06:47:33.740 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_started cluster1 mirror image_11 2024-09-12T06:47:33.740 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:47:33.740 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:47:33.740 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_11 2024-09-12T06:47:33.740 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_state cluster1 mirror image_11 started 2024-09-12T06:47:33.741 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:47:33.741 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:47:33.741 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_11 2024-09-12T06:47:33.741 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=started 2024-09-12T06:47:33.741 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:47:33.741 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:47:33.741 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:47:34.742 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_replay_state cluster1 mirror image_11 started 2024-09-12T06:47:34.742 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:47:34.742 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:47:34.742 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_11 2024-09-12T06:47:34.742 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=started 2024-09-12T06:47:34.743 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local status_result 2024-09-12T06:47:34.743 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=stopped 2024-09-12T06:47:34.743 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_11 2024-09-12T06:47:34.743 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_instance=cluster1 2024-09-12T06:47:34.743 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:47:34.743 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -i state 2024-09-12T06:47:34.743 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:47:34.743 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance=cluster1 2024-09-12T06:47:34.743 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local loop_instance 2024-09-12T06:47:34.744 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:47:34.744 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sleep 0 2024-09-12T06:47:34.744 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:47:34.744 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ seq 0 1 2024-09-12T06:47:34.747 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:47:34.747 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_11 2024-09-12T06:47:34.747 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:0 2024-09-12T06:47:34.747 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:47:34.748 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:47:34.748 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:47:34.748 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:47:34.748 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:47:34.748 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:47:34.748 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:47:34.748 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:47:34.748 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:47:34.748 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:47:34.748 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:47:34.748 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:47:34.748 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:47:34.749 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:47:34.749 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:0 2024-09-12T06:47:34.749 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:47:34.749 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:47:34.749 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:47:34.749 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:0 2024-09-12T06:47:34.749 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:47:34.749 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:47:34.749 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:47:34.749 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:47:34.749 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T06:47:34.749 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:47:34.749 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:47:34.749 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=0 2024-09-12T06:47:34.750 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=0 2024-09-12T06:47:34.750 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:47:34.763 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:47:34.764 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:47:34.764 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:47:34.764 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_11 2024-09-12T06:47:34.887 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ return 0 2024-09-12T06:47:34.888 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T06:47:34.888 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo ' "state": "Replaying"' 2024-09-12T06:47:34.888 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep -i Replaying 2024-09-12T06:47:34.888 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: "state": "Replaying" 2024-09-12T06:47:34.890 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_state=started 2024-09-12T06:47:34.890 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test started = started 2024-09-12T06:47:34.890 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:47:34.891 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_11 2024-09-12T06:47:34.891 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:47:34.891 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:47:34.891 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:47:34.891 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:47:34.891 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_11 2024-09-12T06:47:34.891 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' journal = journal ']' 2024-09-12T06:47:34.891 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror image_11 2024-09-12T06:47:34.891 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:47:34.891 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:47:34.891 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:47:34.891 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:47:34.891 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_11 2024-09-12T06:47:34.891 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-12T06:47:34.892 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-12T06:47:34.892 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:47:34.892 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:47:34.892 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:47:35.096 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror image_11 2024-09-12T06:47:35.096 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:47:35.096 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:47:35.096 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_11 2024-09-12T06:47:35.096 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:47:35.097 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n image_11 ']' 2024-09-12T06:47:35.097 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/image_11' 2024-09-12T06:47:35.097 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_11 2024-09-12T06:47:35.097 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:47:35.097 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:47:35.097 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:47:35.097 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:47:35.097 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:47:35.098 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:47:35.098 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:47:35.098 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:47:35.098 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:47:35.098 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:47:35.098 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_11 2024-09-12T06:47:35.098 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:47:35.098 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:47:35.098 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:47:35.099 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:47:35.099 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:47:35.099 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:47:35.099 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:47:35.099 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:47:35.099 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:47:35.099 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:47:35.099 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:47:35.099 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:47:35.099 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:47:35.100 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:47:35.100 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:47:35.100 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:47:35.100 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:47:35.100 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:47:35.100 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:47:35.100 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:47:35.100 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:47:35.100 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:47:35.100 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:47:35.100 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:47:35.100 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:47:35.101 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:47:35.101 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:47:35.101 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:47:35.101 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:47:35.101 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:47:35.113 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:47:35.113 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:47:35.113 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:47:35.113 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_11 2024-09-12T06:47:35.211 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:47:35.211 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror image_11 2024-09-12T06:47:35.211 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:47:35.211 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:47:35.211 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_11 2024-09-12T06:47:35.212 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_11 '' 2024-09-12T06:47:35.212 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:47:35.212 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:47:35.212 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_11 2024-09-12T06:47:35.212 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:47:35.212 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_11.status 2024-09-12T06:47:35.212 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_11.status 2024-09-12T06:47:35.212 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:47:35.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_11.status 2024-09-12T06:47:35.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_11 2024-09-12T06:47:35.214 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_11.status 2024-09-12T06:47:35.257 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 1 2024-09-12T06:47:35.258 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 1 2024-09-12T06:47:35.258 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:47:35.258 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=5298], [object_number=7, tag_tid=1, entry_tid=5295]]], state=connected] 2024-09-12T06:47:35.258 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=5298], [object_number=7, tag_tid=1, entry_tid=5295]]], state=connected] 2024-09-12T06:47:35.260 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_11.status 2024-09-12T06:47:35.262 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-09-12T06:47:35.263 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror image_11 2024-09-12T06:47:35.263 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:47:35.263 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:47:35.264 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_11 2024-09-12T06:47:35.264 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_11 '..*' 2024-09-12T06:47:35.264 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:47:35.264 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:47:35.264 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_11 2024-09-12T06:47:35.264 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:47:35.264 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_11.status 2024-09-12T06:47:35.264 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_11.status 2024-09-12T06:47:35.264 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:47:35.265 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_11.status 2024-09-12T06:47:35.265 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_11 2024-09-12T06:47:35.265 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_11.status 2024-09-12T06:47:35.310 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 1 2024-09-12T06:47:35.310 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 1 2024-09-12T06:47:35.310 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:47:35.311 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=5298], [object_number=7, tag_tid=1, entry_tid=5295]]], state=connected] 2024-09-12T06:47:35.311 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=5298], [object_number=7, tag_tid=1, entry_tid=5295]]], state=connected] 2024-09-12T06:47:35.313 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_11.status 2024-09-12T06:47:35.315 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-09-12T06:47:35.315 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-09-12T06:47:35.316 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:47:35.316 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_snap_present cluster1 mirror image_11 snap 2024-09-12T06:47:35.316 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:47:35.316 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:47:35.316 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_11 2024-09-12T06:47:35.316 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:47:35.316 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:47:35.316 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T06:47:35.316 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:47:36.316 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 info mirror/image_11@snap 2024-09-12T06:47:36.364 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:rbd image 'image_11': 2024-09-12T06:47:36.364 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: size 128 MiB in 32 objects 2024-09-12T06:47:36.365 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: order 22 (4 MiB objects) 2024-09-12T06:47:36.365 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: snapshot_count: 1 2024-09-12T06:47:36.365 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: id: 10a8d140e7d7 2024-09-12T06:47:36.365 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: block_name_prefix: rbd_data.10a8d140e7d7 2024-09-12T06:47:36.365 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: format: 2 2024-09-12T06:47:36.365 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: features: layering, exclusive-lock, journaling 2024-09-12T06:47:36.365 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: op_features: 2024-09-12T06:47:36.365 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: flags: 2024-09-12T06:47:36.365 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: create_timestamp: Thu Sep 12 06:40:09 2024 2024-09-12T06:47:36.365 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: access_timestamp: Thu Sep 12 06:40:09 2024 2024-09-12T06:47:36.366 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: modify_timestamp: Thu Sep 12 06:40:09 2024 2024-09-12T06:47:36.366 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: protected: False 2024-09-12T06:47:36.366 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: journal: 10a8d140e7d7 2024-09-12T06:47:36.366 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring state: enabled 2024-09-12T06:47:36.366 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring mode: journal 2024-09-12T06:47:36.366 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring global id: 212c933e-f92f-4afc-9a13-6cf37b75e1bb 2024-09-12T06:47:36.366 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring primary: false 2024-09-12T06:47:36.368 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:47:36.368 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror image_11 snap 2024-09-12T06:47:36.368 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:47:36.368 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_11 2024-09-12T06:47:36.368 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:47:36.368 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:47:36.368 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_11.export 2024-09-12T06:47:36.369 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_11.export 2024-09-12T06:47:36.369 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_11.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_11.export 2024-09-12T06:47:36.369 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export image_11@snap /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_11.export 2024-09-12T06:47:36.509 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:47:36.517 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export image_11@snap /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_11.export 2024-09-12T06:47:36.691 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:47:36.698 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_11.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_11.export 2024-09-12T06:47:36.754 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_11.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_11.export 2024-09-12T06:47:36.788 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:47:36.789 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:47:36.789 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_12 2024-09-12T06:47:36.789 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_snap cluster2 mirror image_12 snap 2024-09-12T06:47:36.789 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:47:36.789 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:47:36.789 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_12 2024-09-12T06:47:36.789 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:47:36.789 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror snap create image_12@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T06:47:38.756 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_started cluster1 mirror image_12 2024-09-12T06:47:38.757 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:47:38.757 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:47:38.757 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_12 2024-09-12T06:47:38.757 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_state cluster1 mirror image_12 started 2024-09-12T06:47:38.757 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:47:38.757 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:47:38.757 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_12 2024-09-12T06:47:38.757 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=started 2024-09-12T06:47:38.757 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:47:38.757 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:47:38.757 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:47:39.758 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_replay_state cluster1 mirror image_12 started 2024-09-12T06:47:39.758 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:47:39.759 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:47:39.759 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_12 2024-09-12T06:47:39.759 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=started 2024-09-12T06:47:39.759 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local status_result 2024-09-12T06:47:39.759 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=stopped 2024-09-12T06:47:39.759 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_12 2024-09-12T06:47:39.759 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_instance=cluster1 2024-09-12T06:47:39.759 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:47:39.759 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:47:39.760 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -i state 2024-09-12T06:47:39.760 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance=cluster1 2024-09-12T06:47:39.760 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local loop_instance 2024-09-12T06:47:39.760 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:47:39.760 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sleep 0 2024-09-12T06:47:39.763 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:47:39.764 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ seq 0 1 2024-09-12T06:47:39.764 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:47:39.764 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_12 2024-09-12T06:47:39.764 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:0 2024-09-12T06:47:39.764 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:47:39.764 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:47:39.764 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:47:39.764 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:47:39.765 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:47:39.765 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:47:39.765 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:47:39.765 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:47:39.765 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:47:39.765 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:47:39.765 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:47:39.765 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:47:39.765 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:47:39.765 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:47:39.766 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:0 2024-09-12T06:47:39.766 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:47:39.766 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:47:39.766 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:47:39.766 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:0 2024-09-12T06:47:39.766 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:47:39.766 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:47:39.766 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:47:39.766 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:47:39.766 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T06:47:39.766 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:47:39.766 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:47:39.767 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=0 2024-09-12T06:47:39.767 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=0 2024-09-12T06:47:39.767 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:47:39.788 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:47:39.788 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:47:39.788 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:47:39.788 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_12 2024-09-12T06:47:39.897 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2024-09-12T06:47:39.897 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T06:47:39.897 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:admin_socket: invalid command 2024-09-12T06:47:39.901 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:47:39.901 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_12 2024-09-12T06:47:39.901 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:1 2024-09-12T06:47:39.901 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:47:39.902 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:47:39.902 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T06:47:39.902 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:1 2024-09-12T06:47:39.902 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:47:39.902 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:47:39.902 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:47:39.902 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:47:39.902 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T06:47:39.902 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:47:39.902 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:47:39.902 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=1 2024-09-12T06:47:39.902 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:47:39.902 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T06:47:39.903 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:1 2024-09-12T06:47:39.903 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:47:39.903 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:47:39.903 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T06:47:39.903 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:1 2024-09-12T06:47:39.903 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:47:39.903 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:47:39.903 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:47:39.903 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:47:39.903 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T06:47:39.903 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:47:39.903 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:47:39.903 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=1 2024-09-12T06:47:39.903 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=1 2024-09-12T06:47:39.903 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T06:47:39.919 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:47:39.919 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:47:39.919 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:47:39.919 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_12 2024-09-12T06:47:40.022 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ return 0 2024-09-12T06:47:40.022 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T06:47:40.022 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo ' "state": "Replaying"' 2024-09-12T06:47:40.022 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep -i Replaying 2024-09-12T06:47:40.023 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: "state": "Replaying" 2024-09-12T06:47:40.025 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_state=started 2024-09-12T06:47:40.026 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test started = started 2024-09-12T06:47:40.026 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:47:40.026 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_12 2024-09-12T06:47:40.026 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:47:40.026 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:47:40.026 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:47:40.026 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:47:40.026 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_12 2024-09-12T06:47:40.026 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' journal = journal ']' 2024-09-12T06:47:40.026 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror image_12 2024-09-12T06:47:40.027 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:47:40.028 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:47:40.028 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:47:40.028 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:47:40.028 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_12 2024-09-12T06:47:40.028 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-12T06:47:40.028 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-12T06:47:40.028 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:47:40.028 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:47:40.028 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:47:40.225 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror image_12 2024-09-12T06:47:40.226 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:47:40.226 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:47:40.226 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_12 2024-09-12T06:47:40.226 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:47:40.226 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n image_12 ']' 2024-09-12T06:47:40.226 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/image_12' 2024-09-12T06:47:40.226 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_12 2024-09-12T06:47:40.226 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:47:40.226 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:47:40.226 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:47:40.226 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:47:40.226 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:47:40.226 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:47:40.226 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:47:40.227 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:47:40.227 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:47:40.229 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:47:40.229 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_12 2024-09-12T06:47:40.229 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:47:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:47:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:47:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:47:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:47:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:47:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:47:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:47:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:47:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:47:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:47:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:47:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:47:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:47:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:47:40.231 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:47:40.231 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:47:40.231 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:47:40.231 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:47:40.231 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:47:40.231 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:47:40.231 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:47:40.231 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:47:40.231 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:47:40.231 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:47:40.231 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:47:40.231 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:47:40.231 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:47:40.231 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:47:40.232 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:47:40.247 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:47:40.247 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:47:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:47:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_12 2024-09-12T06:47:40.333 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2024-09-12T06:47:40.333 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd mirror flush : flush rbd mirror 2024-09-12T06:47:40.333 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:admin_socket: invalid command 2024-09-12T06:47:40.337 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:47:40.337 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_12 2024-09-12T06:47:40.337 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:1 2024-09-12T06:47:40.337 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:47:40.337 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:47:40.338 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-12T06:47:40.338 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:1 2024-09-12T06:47:40.338 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:47:40.338 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:47:40.338 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:47:40.338 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=1 2024-09-12T06:47:40.338 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-12T06:47:40.338 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:47:40.338 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:47:40.338 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=1 2024-09-12T06:47:40.338 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:47:40.338 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-12T06:47:40.338 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:1 2024-09-12T06:47:40.338 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:47:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:47:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T06:47:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:1 2024-09-12T06:47:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:47:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:47:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:47:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:47:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T06:47:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:47:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:47:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=1 2024-09-12T06:47:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:47:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T06:47:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:47:40.357 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:47:40.357 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:47:40.357 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_12 2024-09-12T06:47:40.452 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:47:40.453 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror image_12 2024-09-12T06:47:40.454 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:47:40.454 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:47:40.454 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_12 2024-09-12T06:47:40.454 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_12 '' 2024-09-12T06:47:40.454 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:47:40.454 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:47:40.454 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_12 2024-09-12T06:47:40.454 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:47:40.454 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_12.status 2024-09-12T06:47:40.454 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_12.status 2024-09-12T06:47:40.454 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:47:40.455 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_12.status 2024-09-12T06:47:40.455 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_12 2024-09-12T06:47:40.455 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_12.status 2024-09-12T06:47:40.501 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 1 2024-09-12T06:47:40.501 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 1 2024-09-12T06:47:40.501 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:47:40.501 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=4810], [object_number=7, tag_tid=1, entry_tid=4807]]], state=connected] 2024-09-12T06:47:40.501 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=4810], [object_number=7, tag_tid=1, entry_tid=4807]]], state=connected] 2024-09-12T06:47:40.504 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_12.status 2024-09-12T06:47:40.505 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-09-12T06:47:40.507 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror image_12 2024-09-12T06:47:40.507 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:47:40.507 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:47:40.507 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_12 2024-09-12T06:47:40.507 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_12 '..*' 2024-09-12T06:47:40.508 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:47:40.508 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:47:40.508 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_12 2024-09-12T06:47:40.508 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:47:40.508 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_12.status 2024-09-12T06:47:40.508 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_12.status 2024-09-12T06:47:40.508 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:47:40.509 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_12.status 2024-09-12T06:47:40.509 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_12 2024-09-12T06:47:40.509 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_12.status 2024-09-12T06:47:40.553 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 1 2024-09-12T06:47:40.553 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 1 2024-09-12T06:47:40.553 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:47:40.553 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=4810], [object_number=7, tag_tid=1, entry_tid=4807]]], state=connected] 2024-09-12T06:47:40.553 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=4810], [object_number=7, tag_tid=1, entry_tid=4807]]], state=connected] 2024-09-12T06:47:40.556 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_12.status 2024-09-12T06:47:40.558 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-09-12T06:47:40.558 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-09-12T06:47:40.558 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:47:40.558 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_snap_present cluster1 mirror image_12 snap 2024-09-12T06:47:40.558 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:47:40.558 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:47:40.558 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_12 2024-09-12T06:47:40.559 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:47:40.559 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:47:40.559 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T06:47:40.559 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:47:41.559 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 info mirror/image_12@snap 2024-09-12T06:47:41.604 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:rbd image 'image_12': 2024-09-12T06:47:41.604 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: size 128 MiB in 32 objects 2024-09-12T06:47:41.604 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: order 22 (4 MiB objects) 2024-09-12T06:47:41.604 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: snapshot_count: 1 2024-09-12T06:47:41.604 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: id: 10af67e66bff 2024-09-12T06:47:41.604 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: block_name_prefix: rbd_data.10af67e66bff 2024-09-12T06:47:41.604 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: format: 2 2024-09-12T06:47:41.604 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: features: layering, exclusive-lock, journaling 2024-09-12T06:47:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: op_features: 2024-09-12T06:47:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: flags: 2024-09-12T06:47:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: create_timestamp: Thu Sep 12 06:40:14 2024 2024-09-12T06:47:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: access_timestamp: Thu Sep 12 06:40:14 2024 2024-09-12T06:47:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: modify_timestamp: Thu Sep 12 06:40:14 2024 2024-09-12T06:47:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: protected: False 2024-09-12T06:47:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: journal: 10af67e66bff 2024-09-12T06:47:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring state: enabled 2024-09-12T06:47:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring mode: journal 2024-09-12T06:47:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring global id: 4a929751-fc07-4339-aec6-579b05e5fc98 2024-09-12T06:47:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring primary: false 2024-09-12T06:47:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:47:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror image_12 snap 2024-09-12T06:47:41.608 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:47:41.608 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_12 2024-09-12T06:47:41.608 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:47:41.608 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:47:41.608 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_12.export 2024-09-12T06:47:41.608 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_12.export 2024-09-12T06:47:41.608 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_12.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_12.export 2024-09-12T06:47:41.608 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export image_12@snap /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_12.export 2024-09-12T06:47:41.750 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:47:41.757 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export image_12@snap /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_12.export 2024-09-12T06:47:41.939 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:47:41.945 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_12.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_12.export 2024-09-12T06:47:42.001 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_12.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_12.export 2024-09-12T06:47:42.036 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:47:42.037 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:47:42.037 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_13 2024-09-12T06:47:42.037 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_snap cluster2 mirror image_13 snap 2024-09-12T06:47:42.037 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:47:42.037 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:47:42.037 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_13 2024-09-12T06:47:42.037 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:47:42.037 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror snap create image_13@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T06:47:43.757 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_started cluster1 mirror image_13 2024-09-12T06:47:43.758 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:47:43.758 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:47:43.758 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_13 2024-09-12T06:47:43.758 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_state cluster1 mirror image_13 started 2024-09-12T06:47:43.758 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:47:43.758 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:47:43.758 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_13 2024-09-12T06:47:43.758 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=started 2024-09-12T06:47:43.758 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:47:43.758 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:47:43.758 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:47:44.758 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_replay_state cluster1 mirror image_13 started 2024-09-12T06:47:44.759 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:47:44.759 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:47:44.759 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_13 2024-09-12T06:47:44.759 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=started 2024-09-12T06:47:44.759 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local status_result 2024-09-12T06:47:44.759 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=stopped 2024-09-12T06:47:44.760 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_13 2024-09-12T06:47:44.760 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_instance=cluster1 2024-09-12T06:47:44.760 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -i state 2024-09-12T06:47:44.760 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:47:44.760 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:47:44.761 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance=cluster1 2024-09-12T06:47:44.761 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local loop_instance 2024-09-12T06:47:44.761 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:47:44.761 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sleep 0 2024-09-12T06:47:44.761 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:47:44.761 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ seq 0 1 2024-09-12T06:47:44.762 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:47:44.762 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_13 2024-09-12T06:47:44.763 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:0 2024-09-12T06:47:44.763 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:47:44.763 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:47:44.763 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:47:44.763 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:47:44.763 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:47:44.763 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:47:44.763 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:47:44.763 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:47:44.763 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:47:44.763 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:47:44.763 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:47:44.763 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:47:44.763 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:47:44.765 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:47:44.765 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:0 2024-09-12T06:47:44.765 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:47:44.765 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:47:44.766 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:47:44.766 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:0 2024-09-12T06:47:44.766 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:47:44.766 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:47:44.766 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:47:44.766 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:47:44.766 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T06:47:44.766 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:47:44.766 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:47:44.766 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=0 2024-09-12T06:47:44.766 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=0 2024-09-12T06:47:44.766 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:47:44.790 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:47:44.790 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:47:44.790 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:47:44.790 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_13 2024-09-12T06:47:44.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ return 0 2024-09-12T06:47:44.907 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T06:47:44.907 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo ' "state": "Replaying"' 2024-09-12T06:47:44.907 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep -i Replaying 2024-09-12T06:47:44.908 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: "state": "Replaying" 2024-09-12T06:47:44.910 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_state=started 2024-09-12T06:47:44.910 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test started = started 2024-09-12T06:47:44.910 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:47:44.910 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_13 2024-09-12T06:47:44.910 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:47:44.910 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:47:44.910 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:47:44.910 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:47:44.910 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_13 2024-09-12T06:47:44.910 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' journal = journal ']' 2024-09-12T06:47:44.910 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror image_13 2024-09-12T06:47:44.910 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:47:44.911 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:47:44.911 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:47:44.911 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:47:44.911 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_13 2024-09-12T06:47:44.911 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-12T06:47:44.911 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-12T06:47:44.911 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:47:44.911 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:47:44.911 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:47:45.111 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror image_13 2024-09-12T06:47:45.111 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:47:45.111 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:47:45.111 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_13 2024-09-12T06:47:45.111 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:47:45.111 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n image_13 ']' 2024-09-12T06:47:45.111 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/image_13' 2024-09-12T06:47:45.111 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_13 2024-09-12T06:47:45.111 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:47:45.111 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:47:45.112 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:47:45.112 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:47:45.112 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:47:45.112 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:47:45.112 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:47:45.112 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:47:45.112 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:47:45.115 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:47:45.115 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_13 2024-09-12T06:47:45.115 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:47:45.115 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:47:45.115 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:47:45.115 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:47:45.116 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:47:45.116 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:47:45.116 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:47:45.116 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:47:45.116 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:47:45.116 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:47:45.116 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:47:45.116 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:47:45.116 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:47:45.116 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:47:45.116 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:47:45.116 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:47:45.116 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:47:45.117 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:47:45.117 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:47:45.117 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:47:45.117 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:47:45.117 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:47:45.117 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:47:45.117 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:47:45.117 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:47:45.117 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:47:45.117 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:47:45.117 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:47:45.117 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:47:45.117 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:47:45.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:47:45.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:47:45.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:47:45.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_13 2024-09-12T06:47:45.228 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:47:45.228 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror image_13 2024-09-12T06:47:45.228 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:47:45.228 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:47:45.228 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_13 2024-09-12T06:47:45.228 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_13 '' 2024-09-12T06:47:45.228 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:47:45.229 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:47:45.229 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_13 2024-09-12T06:47:45.229 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:47:45.229 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_13.status 2024-09-12T06:47:45.229 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_13.status 2024-09-12T06:47:45.229 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:47:45.229 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_13.status 2024-09-12T06:47:45.230 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_13 2024-09-12T06:47:45.230 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_13.status 2024-09-12T06:47:45.276 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 3 2024-09-12T06:47:45.276 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 4 2024-09-12T06:47:45.276 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:47:45.276 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=13507], [object_number=14, tag_tid=1, entry_tid=13506]]], state=connected] 2024-09-12T06:47:45.276 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=13507], [object_number=14, tag_tid=1, entry_tid=13506]]], state=connected] 2024-09-12T06:47:45.280 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_13.status 2024-09-12T06:47:45.281 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=17, tag_tid=2, entry_tid=1' 2024-09-12T06:47:45.281 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror image_13 2024-09-12T06:47:45.281 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:47:45.282 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:47:45.282 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_13 2024-09-12T06:47:45.282 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_13 '..*' 2024-09-12T06:47:45.282 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:47:45.282 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:47:45.282 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_13 2024-09-12T06:47:45.282 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:47:45.282 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_13.status 2024-09-12T06:47:45.282 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_13.status 2024-09-12T06:47:45.282 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:47:45.283 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_13.status 2024-09-12T06:47:45.283 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_13 2024-09-12T06:47:45.283 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_13.status 2024-09-12T06:47:45.328 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 3 2024-09-12T06:47:45.328 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 4 2024-09-12T06:47:45.328 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:47:45.328 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=13507], [object_number=14, tag_tid=1, entry_tid=13506]]], state=connected] 2024-09-12T06:47:45.328 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=13507], [object_number=14, tag_tid=1, entry_tid=13506]]], state=connected] 2024-09-12T06:47:45.331 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_13.status 2024-09-12T06:47:45.333 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=17, tag_tid=2, entry_tid=1' 2024-09-12T06:47:45.333 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=17, tag_tid=2, entry_tid=1' -a 'object_number=17, tag_tid=2, entry_tid=1' = 'object_number=17, tag_tid=2, entry_tid=1' 2024-09-12T06:47:45.333 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:47:45.333 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_snap_present cluster1 mirror image_13 snap 2024-09-12T06:47:45.333 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:47:45.333 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:47:45.333 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_13 2024-09-12T06:47:45.334 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:47:45.334 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:47:45.334 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T06:47:45.334 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:47:46.334 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 info mirror/image_13@snap 2024-09-12T06:47:46.383 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:rbd image 'image_13': 2024-09-12T06:47:46.383 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: size 128 MiB in 32 objects 2024-09-12T06:47:46.383 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: order 22 (4 MiB objects) 2024-09-12T06:47:46.383 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: snapshot_count: 1 2024-09-12T06:47:46.383 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: id: 10a84c139f89 2024-09-12T06:47:46.384 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: block_name_prefix: rbd_data.10a84c139f89 2024-09-12T06:47:46.384 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: format: 2 2024-09-12T06:47:46.384 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: features: layering, exclusive-lock, journaling 2024-09-12T06:47:46.384 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: op_features: 2024-09-12T06:47:46.384 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: flags: 2024-09-12T06:47:46.384 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: create_timestamp: Thu Sep 12 06:40:21 2024 2024-09-12T06:47:46.384 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: access_timestamp: Thu Sep 12 06:40:21 2024 2024-09-12T06:47:46.384 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: modify_timestamp: Thu Sep 12 06:41:22 2024 2024-09-12T06:47:46.384 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: protected: False 2024-09-12T06:47:46.384 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: journal: 10a84c139f89 2024-09-12T06:47:46.384 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring state: enabled 2024-09-12T06:47:46.384 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring mode: journal 2024-09-12T06:47:46.384 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring global id: 857fe2a7-0031-49de-8ea5-162e1725ee55 2024-09-12T06:47:46.384 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring primary: false 2024-09-12T06:47:46.387 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:47:46.387 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror image_13 snap 2024-09-12T06:47:46.387 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:47:46.387 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_13 2024-09-12T06:47:46.387 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:47:46.388 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:47:46.388 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_13.export 2024-09-12T06:47:46.388 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_13.export 2024-09-12T06:47:46.388 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_13.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_13.export 2024-09-12T06:47:46.388 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export image_13@snap /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_13.export 2024-09-12T06:47:46.546 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:47:46.554 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export image_13@snap /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_13.export 2024-09-12T06:47:46.768 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:47:46.775 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_13.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_13.export 2024-09-12T06:47:46.830 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_13.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_13.export 2024-09-12T06:47:46.863 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:47:46.863 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:47:46.863 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_14 2024-09-12T06:47:46.863 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_snap cluster2 mirror image_14 snap 2024-09-12T06:47:46.863 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:47:46.863 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:47:46.863 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_14 2024-09-12T06:47:46.863 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:47:46.863 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror snap create image_14@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T06:47:49.787 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_started cluster1 mirror image_14 2024-09-12T06:47:49.787 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:47:49.787 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:47:49.787 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_14 2024-09-12T06:47:49.787 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_state cluster1 mirror image_14 started 2024-09-12T06:47:49.787 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:47:49.787 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:47:49.787 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_14 2024-09-12T06:47:49.787 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=started 2024-09-12T06:47:49.787 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:47:49.788 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:47:49.788 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:47:50.789 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_replay_state cluster1 mirror image_14 started 2024-09-12T06:47:50.789 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:47:50.789 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:47:50.789 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_14 2024-09-12T06:47:50.789 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=started 2024-09-12T06:47:50.789 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local status_result 2024-09-12T06:47:50.789 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=stopped 2024-09-12T06:47:50.789 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_14 2024-09-12T06:47:50.790 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_instance=cluster1 2024-09-12T06:47:50.790 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:47:50.790 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -i state 2024-09-12T06:47:50.790 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:47:50.790 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance=cluster1 2024-09-12T06:47:50.790 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local loop_instance 2024-09-12T06:47:50.790 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:47:50.790 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sleep 0 2024-09-12T06:47:50.790 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:47:50.792 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ seq 0 1 2024-09-12T06:47:50.792 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:47:50.792 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_14 2024-09-12T06:47:50.792 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:0 2024-09-12T06:47:50.792 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:47:50.792 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:47:50.792 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:47:50.793 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:47:50.793 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:47:50.793 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:47:50.793 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:47:50.793 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:47:50.793 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:47:50.793 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:47:50.795 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:47:50.795 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:47:50.795 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:47:50.796 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:47:50.796 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:0 2024-09-12T06:47:50.796 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:47:50.796 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:47:50.796 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:47:50.796 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:0 2024-09-12T06:47:50.796 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:47:50.796 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:47:50.796 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:47:50.796 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:47:50.796 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T06:47:50.796 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:47:50.797 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:47:50.797 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=0 2024-09-12T06:47:50.797 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=0 2024-09-12T06:47:50.797 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:47:50.811 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:47:50.812 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:47:50.812 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:47:50.812 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_14 2024-09-12T06:47:50.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2024-09-12T06:47:50.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T06:47:50.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:admin_socket: invalid command 2024-09-12T06:47:50.909 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:47:50.910 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_14 2024-09-12T06:47:50.910 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:1 2024-09-12T06:47:50.910 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:47:50.910 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:47:50.910 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T06:47:50.910 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:1 2024-09-12T06:47:50.910 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:47:50.910 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:47:50.910 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:47:50.910 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:47:50.910 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T06:47:50.910 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:47:50.911 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:47:50.911 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=1 2024-09-12T06:47:50.911 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:47:50.911 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T06:47:50.911 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:1 2024-09-12T06:47:50.911 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:47:50.911 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:47:50.911 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T06:47:50.911 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:1 2024-09-12T06:47:50.911 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:47:50.911 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:47:50.911 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:47:50.912 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:47:50.912 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T06:47:50.912 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:47:50.912 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:47:50.912 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=1 2024-09-12T06:47:50.912 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=1 2024-09-12T06:47:50.912 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T06:47:50.928 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:47:50.928 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:47:50.928 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:47:50.928 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_14 2024-09-12T06:47:51.029 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ return 0 2024-09-12T06:47:51.030 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T06:47:51.030 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo ' "state": "Replaying"' 2024-09-12T06:47:51.030 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep -i Replaying 2024-09-12T06:47:51.031 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: "state": "Replaying" 2024-09-12T06:47:51.033 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_state=started 2024-09-12T06:47:51.034 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test started = started 2024-09-12T06:47:51.034 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:47:51.034 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_14 2024-09-12T06:47:51.034 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:47:51.034 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:47:51.034 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:47:51.034 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:47:51.034 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_14 2024-09-12T06:47:51.035 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' journal = journal ']' 2024-09-12T06:47:51.035 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror image_14 2024-09-12T06:47:51.035 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:47:51.035 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:47:51.035 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:47:51.035 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:47:51.035 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_14 2024-09-12T06:47:51.035 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-12T06:47:51.036 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-12T06:47:51.036 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:47:51.036 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:47:51.036 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:47:51.233 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror image_14 2024-09-12T06:47:51.233 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:47:51.234 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:47:51.234 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_14 2024-09-12T06:47:51.234 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:47:51.234 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n image_14 ']' 2024-09-12T06:47:51.234 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/image_14' 2024-09-12T06:47:51.234 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_14 2024-09-12T06:47:51.234 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:47:51.234 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:47:51.234 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:47:51.234 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:47:51.234 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:47:51.234 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:47:51.234 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:47:51.234 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:47:51.235 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:47:51.238 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:47:51.238 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_14 2024-09-12T06:47:51.238 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:47:51.238 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:47:51.238 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:47:51.238 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:47:51.239 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:47:51.239 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:47:51.239 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:47:51.239 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:47:51.239 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:47:51.239 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:47:51.239 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:47:51.239 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:47:51.239 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:47:51.239 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:47:51.239 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:47:51.239 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:47:51.239 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:47:51.239 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:47:51.240 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:47:51.240 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:47:51.240 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:47:51.240 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:47:51.240 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:47:51.240 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:47:51.240 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:47:51.240 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:47:51.240 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:47:51.240 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:47:51.240 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:47:51.240 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:47:51.254 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:47:51.255 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:47:51.255 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:47:51.255 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_14 2024-09-12T06:47:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2024-09-12T06:47:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd mirror flush : flush rbd mirror 2024-09-12T06:47:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:admin_socket: invalid command 2024-09-12T06:47:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:47:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_14 2024-09-12T06:47:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:1 2024-09-12T06:47:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:47:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:47:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-12T06:47:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:1 2024-09-12T06:47:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:47:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:47:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:47:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=1 2024-09-12T06:47:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-12T06:47:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:47:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:47:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=1 2024-09-12T06:47:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:47:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-12T06:47:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:1 2024-09-12T06:47:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:47:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:47:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T06:47:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:1 2024-09-12T06:47:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:47:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:47:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:47:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:47:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T06:47:51.347 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:47:51.347 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:47:51.347 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=1 2024-09-12T06:47:51.347 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:47:51.347 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T06:47:51.364 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:47:51.364 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:47:51.364 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:47:51.364 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_14 2024-09-12T06:47:51.463 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:47:51.463 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror image_14 2024-09-12T06:47:51.463 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:47:51.463 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:47:51.464 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_14 2024-09-12T06:47:51.464 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_14 '' 2024-09-12T06:47:51.464 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:47:51.464 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:47:51.464 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_14 2024-09-12T06:47:51.464 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:47:51.464 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_14.status 2024-09-12T06:47:51.464 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_14.status 2024-09-12T06:47:51.465 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:47:51.465 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_14.status 2024-09-12T06:47:51.465 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_14 2024-09-12T06:47:51.466 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_14.status 2024-09-12T06:47:51.509 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 2 2024-09-12T06:47:51.509 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 2 2024-09-12T06:47:51.509 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:47:51.509 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=7959], [object_number=10, tag_tid=1, entry_tid=7958]]], state=connected] 2024-09-12T06:47:51.510 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=7959], [object_number=10, tag_tid=1, entry_tid=7958]]], state=connected] 2024-09-12T06:47:51.512 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_14.status 2024-09-12T06:47:51.513 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-09-12T06:47:51.515 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror image_14 2024-09-12T06:47:51.515 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:47:51.515 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:47:51.515 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_14 2024-09-12T06:47:51.515 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_14 '..*' 2024-09-12T06:47:51.516 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:47:51.516 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:47:51.516 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_14 2024-09-12T06:47:51.516 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:47:51.516 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_14.status 2024-09-12T06:47:51.516 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_14.status 2024-09-12T06:47:51.516 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:47:51.517 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_14.status 2024-09-12T06:47:51.517 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_14 2024-09-12T06:47:51.518 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_14.status 2024-09-12T06:47:51.561 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 2 2024-09-12T06:47:51.561 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 2 2024-09-12T06:47:51.561 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:47:51.561 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=7959], [object_number=10, tag_tid=1, entry_tid=7958]]], state=connected] 2024-09-12T06:47:51.562 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=7959], [object_number=10, tag_tid=1, entry_tid=7958]]], state=connected] 2024-09-12T06:47:51.564 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_14.status 2024-09-12T06:47:51.567 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-09-12T06:47:51.567 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-09-12T06:47:51.567 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:47:51.567 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_snap_present cluster1 mirror image_14 snap 2024-09-12T06:47:51.567 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:47:51.567 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:47:51.568 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_14 2024-09-12T06:47:51.568 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:47:51.568 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:47:51.568 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T06:47:51.568 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:47:52.567 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 info mirror/image_14@snap 2024-09-12T06:47:52.621 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:rbd image 'image_14': 2024-09-12T06:47:52.622 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: size 128 MiB in 32 objects 2024-09-12T06:47:52.622 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: order 22 (4 MiB objects) 2024-09-12T06:47:52.622 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: snapshot_count: 1 2024-09-12T06:47:52.622 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: id: 10af2988b211 2024-09-12T06:47:52.622 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: block_name_prefix: rbd_data.10af2988b211 2024-09-12T06:47:52.622 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: format: 2 2024-09-12T06:47:52.623 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: features: layering, exclusive-lock, journaling 2024-09-12T06:47:52.623 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: op_features: 2024-09-12T06:47:52.623 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: flags: 2024-09-12T06:47:52.623 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: create_timestamp: Thu Sep 12 06:40:33 2024 2024-09-12T06:47:52.623 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: access_timestamp: Thu Sep 12 06:40:33 2024 2024-09-12T06:47:52.623 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: modify_timestamp: Thu Sep 12 06:41:34 2024 2024-09-12T06:47:52.624 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: protected: False 2024-09-12T06:47:52.624 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: journal: 10af2988b211 2024-09-12T06:47:52.624 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring state: enabled 2024-09-12T06:47:52.624 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring mode: journal 2024-09-12T06:47:52.624 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring global id: 34ba0457-1a63-45e1-a270-20ae303738d8 2024-09-12T06:47:52.625 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring primary: false 2024-09-12T06:47:52.626 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:47:52.626 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror image_14 snap 2024-09-12T06:47:52.627 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:47:52.627 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_14 2024-09-12T06:47:52.627 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:47:52.627 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:47:52.627 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_14.export 2024-09-12T06:47:52.628 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_14.export 2024-09-12T06:47:52.628 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_14.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_14.export 2024-09-12T06:47:52.628 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export image_14@snap /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_14.export 2024-09-12T06:47:52.777 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:47:52.784 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export image_14@snap /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_14.export 2024-09-12T06:47:52.984 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:47:52.991 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_14.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_14.export 2024-09-12T06:47:53.047 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_14.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_14.export 2024-09-12T06:47:53.078 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:47:53.078 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:47:53.079 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_15 2024-09-12T06:47:53.079 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_snap cluster2 mirror image_15 snap 2024-09-12T06:47:53.079 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:47:53.079 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:47:53.079 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_15 2024-09-12T06:47:53.079 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:47:53.079 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror snap create image_15@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T06:47:53.785 INFO:tasks.ceph.cluster2.osd.2.smithi152.stderr:2024-09-12T06:47:53.787+0000 7f4774fb0640 -1 bdev(0x56163c600c00 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 2 2024-09-12T06:47:53.786 INFO:tasks.ceph.cluster2.osd.2.smithi152.stderr:2024-09-12T06:47:53.788+0000 7f47787b7640 -1 bdev(0x56163c600c00 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 4 2024-09-12T06:47:53.786 INFO:tasks.ceph.cluster2.osd.2.smithi152.stderr:2024-09-12T06:47:53.788+0000 7f47797b9640 -1 bdev(0x56163c600c00 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 5 2024-09-12T06:47:54.795 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_started cluster1 mirror image_15 2024-09-12T06:47:54.795 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:47:54.795 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:47:54.795 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_15 2024-09-12T06:47:54.795 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_state cluster1 mirror image_15 started 2024-09-12T06:47:54.795 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:47:54.795 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:47:54.796 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_15 2024-09-12T06:47:54.796 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=started 2024-09-12T06:47:54.796 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:47:54.796 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:47:54.796 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:47:55.797 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_replay_state cluster1 mirror image_15 started 2024-09-12T06:47:55.797 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:47:55.797 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:47:55.798 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_15 2024-09-12T06:47:55.798 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=started 2024-09-12T06:47:55.798 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local status_result 2024-09-12T06:47:55.798 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=stopped 2024-09-12T06:47:55.798 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_15 2024-09-12T06:47:55.798 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_instance=cluster1 2024-09-12T06:47:55.798 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:47:55.798 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -i state 2024-09-12T06:47:55.798 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:47:55.798 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance=cluster1 2024-09-12T06:47:55.798 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local loop_instance 2024-09-12T06:47:55.798 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:47:55.798 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sleep 0 2024-09-12T06:47:55.800 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:47:55.800 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ seq 0 1 2024-09-12T06:47:55.801 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:47:55.801 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_15 2024-09-12T06:47:55.801 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:0 2024-09-12T06:47:55.801 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:47:55.801 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:47:55.801 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:47:55.801 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:47:55.801 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:47:55.801 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:47:55.801 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:47:55.801 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:47:55.801 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:47:55.801 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:47:55.801 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:47:55.802 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:47:55.802 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:47:55.802 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:47:55.802 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:0 2024-09-12T06:47:55.802 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:47:55.802 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:47:55.802 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:47:55.802 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:0 2024-09-12T06:47:55.802 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:47:55.803 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:47:55.803 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:47:55.803 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:47:55.803 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T06:47:55.803 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:47:55.803 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:47:55.803 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=0 2024-09-12T06:47:55.804 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=0 2024-09-12T06:47:55.804 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:47:55.821 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:47:55.821 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:47:55.821 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:47:55.821 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_15 2024-09-12T06:47:55.921 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ return 0 2024-09-12T06:47:55.921 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T06:47:55.922 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo ' "state": "Replaying"' 2024-09-12T06:47:55.922 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep -i Replaying 2024-09-12T06:47:55.922 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: "state": "Replaying" 2024-09-12T06:47:55.924 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_state=started 2024-09-12T06:47:55.924 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test started = started 2024-09-12T06:47:55.924 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:47:55.924 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_15 2024-09-12T06:47:55.924 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:47:55.924 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:47:55.925 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:47:55.925 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:47:55.925 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_15 2024-09-12T06:47:55.925 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' journal = journal ']' 2024-09-12T06:47:55.925 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror image_15 2024-09-12T06:47:55.925 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:47:55.925 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:47:55.925 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:47:55.925 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:47:55.925 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_15 2024-09-12T06:47:55.925 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-12T06:47:55.925 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-12T06:47:55.925 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:47:55.925 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:47:55.926 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:47:56.126 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror image_15 2024-09-12T06:47:56.126 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:47:56.126 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:47:56.126 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_15 2024-09-12T06:47:56.126 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:47:56.126 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n image_15 ']' 2024-09-12T06:47:56.127 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/image_15' 2024-09-12T06:47:56.127 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_15 2024-09-12T06:47:56.127 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:47:56.127 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:47:56.127 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:47:56.127 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:47:56.127 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:47:56.127 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:47:56.127 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:47:56.127 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:47:56.127 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:47:56.131 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:47:56.131 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_15 2024-09-12T06:47:56.131 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:47:56.131 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:47:56.131 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:47:56.131 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:47:56.131 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:47:56.131 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:47:56.131 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:47:56.131 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:47:56.131 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:47:56.131 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:47:56.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:47:56.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:47:56.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:47:56.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:47:56.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:47:56.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:47:56.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:47:56.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:47:56.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:47:56.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:47:56.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:47:56.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:47:56.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:47:56.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:47:56.133 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:47:56.133 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:47:56.133 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:47:56.133 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:47:56.133 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:47:56.133 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:47:56.147 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:47:56.147 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:47:56.147 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:47:56.148 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_15 2024-09-12T06:47:56.243 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:47:56.243 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror image_15 2024-09-12T06:47:56.243 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:47:56.243 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:47:56.243 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_15 2024-09-12T06:47:56.243 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_15 '' 2024-09-12T06:47:56.243 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:47:56.244 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:47:56.244 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_15 2024-09-12T06:47:56.244 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:47:56.244 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_15.status 2024-09-12T06:47:56.244 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_15.status 2024-09-12T06:47:56.244 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:47:56.244 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_15.status 2024-09-12T06:47:56.245 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_15 2024-09-12T06:47:56.245 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_15.status 2024-09-12T06:47:56.290 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 2 2024-09-12T06:47:56.291 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 2 2024-09-12T06:47:56.291 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:47:56.291 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8591], [object_number=10, tag_tid=1, entry_tid=8590]]], state=connected] 2024-09-12T06:47:56.291 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8591], [object_number=10, tag_tid=1, entry_tid=8590]]], state=connected] 2024-09-12T06:47:56.293 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_15.status 2024-09-12T06:47:56.294 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-09-12T06:47:56.295 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror image_15 2024-09-12T06:47:56.296 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:47:56.296 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:47:56.296 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_15 2024-09-12T06:47:56.296 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_15 '..*' 2024-09-12T06:47:56.296 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:47:56.296 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:47:56.296 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_15 2024-09-12T06:47:56.296 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:47:56.296 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_15.status 2024-09-12T06:47:56.297 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_15.status 2024-09-12T06:47:56.297 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:47:56.297 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_15.status 2024-09-12T06:47:56.297 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_15 2024-09-12T06:47:56.298 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_15.status 2024-09-12T06:47:56.342 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 2 2024-09-12T06:47:56.342 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 2 2024-09-12T06:47:56.342 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:47:56.342 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8591], [object_number=10, tag_tid=1, entry_tid=8590]]], state=connected] 2024-09-12T06:47:56.342 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8591], [object_number=10, tag_tid=1, entry_tid=8590]]], state=connected] 2024-09-12T06:47:56.345 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_15.status 2024-09-12T06:47:56.348 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-09-12T06:47:56.348 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-09-12T06:47:56.348 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:47:56.348 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_snap_present cluster1 mirror image_15 snap 2024-09-12T06:47:56.348 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:47:56.348 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:47:56.348 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_15 2024-09-12T06:47:56.348 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:47:56.348 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:47:56.348 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T06:47:56.349 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:47:57.348 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 info mirror/image_15@snap 2024-09-12T06:47:57.395 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:rbd image 'image_15': 2024-09-12T06:47:57.395 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: size 128 MiB in 32 objects 2024-09-12T06:47:57.395 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: order 22 (4 MiB objects) 2024-09-12T06:47:57.396 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: snapshot_count: 1 2024-09-12T06:47:57.396 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: id: 10a89d633fd8 2024-09-12T06:47:57.396 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: block_name_prefix: rbd_data.10a89d633fd8 2024-09-12T06:47:57.396 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: format: 2 2024-09-12T06:47:57.396 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: features: layering, exclusive-lock, journaling 2024-09-12T06:47:57.396 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: op_features: 2024-09-12T06:47:57.396 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: flags: 2024-09-12T06:47:57.396 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: create_timestamp: Thu Sep 12 06:40:43 2024 2024-09-12T06:47:57.396 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: access_timestamp: Thu Sep 12 06:40:43 2024 2024-09-12T06:47:57.396 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: modify_timestamp: Thu Sep 12 06:41:44 2024 2024-09-12T06:47:57.396 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: protected: False 2024-09-12T06:47:57.396 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: journal: 10a89d633fd8 2024-09-12T06:47:57.396 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring state: enabled 2024-09-12T06:47:57.397 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring mode: journal 2024-09-12T06:47:57.397 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring global id: 7cf9c367-7d6b-4568-a80d-e026df556f7c 2024-09-12T06:47:57.397 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring primary: false 2024-09-12T06:47:57.399 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:47:57.399 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror image_15 snap 2024-09-12T06:47:57.399 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:47:57.399 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_15 2024-09-12T06:47:57.399 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:47:57.399 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:47:57.399 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_15.export 2024-09-12T06:47:57.400 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_15.export 2024-09-12T06:47:57.400 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_15.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_15.export 2024-09-12T06:47:57.400 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export image_15@snap /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_15.export 2024-09-12T06:47:57.550 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:47:57.557 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export image_15@snap /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_15.export 2024-09-12T06:47:57.789 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:47:57.796 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_15.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_15.export 2024-09-12T06:47:57.850 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_15.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_15.export 2024-09-12T06:47:57.883 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:47:57.883 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:47:57.883 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_16 2024-09-12T06:47:57.883 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_snap cluster2 mirror image_16 snap 2024-09-12T06:47:57.883 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:47:57.884 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:47:57.884 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_16 2024-09-12T06:47:57.884 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:47:57.884 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror snap create image_16@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T06:47:59.812 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_started cluster1 mirror image_16 2024-09-12T06:47:59.812 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:47:59.812 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:47:59.812 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_16 2024-09-12T06:47:59.813 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_state cluster1 mirror image_16 started 2024-09-12T06:47:59.813 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:47:59.813 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:47:59.813 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_16 2024-09-12T06:47:59.813 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=started 2024-09-12T06:47:59.813 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:47:59.813 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:47:59.813 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:48:00.815 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_replay_state cluster1 mirror image_16 started 2024-09-12T06:48:00.815 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:48:00.815 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:00.815 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_16 2024-09-12T06:48:00.815 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=started 2024-09-12T06:48:00.815 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local status_result 2024-09-12T06:48:00.815 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=stopped 2024-09-12T06:48:00.815 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_16 2024-09-12T06:48:00.815 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -i state 2024-09-12T06:48:00.816 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_instance=cluster1 2024-09-12T06:48:00.816 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:48:00.816 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:48:00.816 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance=cluster1 2024-09-12T06:48:00.816 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local loop_instance 2024-09-12T06:48:00.816 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:48:00.816 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sleep 0 2024-09-12T06:48:00.816 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:48:00.818 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ seq 0 1 2024-09-12T06:48:00.818 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:48:00.818 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_16 2024-09-12T06:48:00.818 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:0 2024-09-12T06:48:00.818 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:48:00.818 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:48:00.818 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:48:00.818 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:48:00.819 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:48:00.819 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:48:00.819 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:48:00.819 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:48:00.819 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:48:00.819 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:48:00.819 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:48:00.819 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:48:00.819 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:48:00.819 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:48:00.819 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:0 2024-09-12T06:48:00.821 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:48:00.821 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:48:00.821 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:48:00.821 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:0 2024-09-12T06:48:00.821 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:48:00.821 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:48:00.821 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:48:00.821 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:48:00.821 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T06:48:00.821 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:48:00.822 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:48:00.822 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=0 2024-09-12T06:48:00.822 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=0 2024-09-12T06:48:00.822 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:48:00.838 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:48:00.838 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:48:00.838 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:48:00.838 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_16 2024-09-12T06:48:00.933 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2024-09-12T06:48:00.933 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T06:48:00.933 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:admin_socket: invalid command 2024-09-12T06:48:00.937 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:48:00.937 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_16 2024-09-12T06:48:00.937 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:1 2024-09-12T06:48:00.937 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:48:00.937 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:48:00.937 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T06:48:00.937 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:1 2024-09-12T06:48:00.938 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:48:00.938 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:48:00.938 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:48:00.938 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:48:00.938 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T06:48:00.938 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:48:00.938 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:48:00.938 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=1 2024-09-12T06:48:00.938 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:48:00.938 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T06:48:00.938 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:1 2024-09-12T06:48:00.938 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:48:00.939 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:48:00.939 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T06:48:00.939 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:1 2024-09-12T06:48:00.939 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:48:00.939 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:48:00.939 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:48:00.939 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:48:00.939 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T06:48:00.939 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:48:00.939 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:48:00.939 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=1 2024-09-12T06:48:00.939 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=1 2024-09-12T06:48:00.939 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T06:48:00.955 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:48:00.956 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:48:00.956 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:48:00.956 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_16 2024-09-12T06:48:01.057 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ return 0 2024-09-12T06:48:01.057 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T06:48:01.057 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo ' "state": "Replaying"' 2024-09-12T06:48:01.057 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep -i Replaying 2024-09-12T06:48:01.058 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: "state": "Replaying" 2024-09-12T06:48:01.059 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_state=started 2024-09-12T06:48:01.059 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test started = started 2024-09-12T06:48:01.060 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:48:01.060 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_16 2024-09-12T06:48:01.060 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:48:01.060 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:48:01.060 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:48:01.060 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:48:01.060 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_16 2024-09-12T06:48:01.060 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' journal = journal ']' 2024-09-12T06:48:01.060 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror image_16 2024-09-12T06:48:01.060 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:48:01.060 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:48:01.060 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:48:01.060 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:48:01.061 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_16 2024-09-12T06:48:01.061 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-12T06:48:01.061 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-12T06:48:01.061 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:48:01.061 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:48:01.061 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:48:01.261 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror image_16 2024-09-12T06:48:01.261 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:48:01.261 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:01.261 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_16 2024-09-12T06:48:01.261 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:48:01.261 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n image_16 ']' 2024-09-12T06:48:01.261 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/image_16' 2024-09-12T06:48:01.261 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_16 2024-09-12T06:48:01.262 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:48:01.262 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:48:01.262 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:48:01.262 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:48:01.262 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:48:01.262 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:48:01.262 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:48:01.262 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:48:01.262 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:48:01.265 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:48:01.265 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_16 2024-09-12T06:48:01.265 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:48:01.265 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:48:01.266 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:48:01.266 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:48:01.266 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:48:01.266 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:48:01.266 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:48:01.266 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:48:01.266 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:48:01.266 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:48:01.266 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:48:01.266 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:48:01.266 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:48:01.266 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:48:01.266 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:48:01.267 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:48:01.267 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:48:01.267 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:48:01.267 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:48:01.267 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:48:01.267 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:48:01.267 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:48:01.267 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:48:01.267 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:48:01.267 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:48:01.267 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:48:01.268 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:48:01.268 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:48:01.268 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:48:01.268 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:48:01.282 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:48:01.282 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:48:01.282 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:48:01.283 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_16 2024-09-12T06:48:01.369 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2024-09-12T06:48:01.369 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd mirror flush : flush rbd mirror 2024-09-12T06:48:01.372 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:admin_socket: invalid command 2024-09-12T06:48:01.376 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:48:01.376 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_16 2024-09-12T06:48:01.376 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:1 2024-09-12T06:48:01.376 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:48:01.376 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:48:01.376 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-12T06:48:01.376 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:1 2024-09-12T06:48:01.376 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:48:01.377 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:48:01.377 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:48:01.377 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=1 2024-09-12T06:48:01.377 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-12T06:48:01.377 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:48:01.377 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:48:01.377 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=1 2024-09-12T06:48:01.377 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:48:01.377 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-12T06:48:01.378 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:1 2024-09-12T06:48:01.378 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:48:01.378 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:48:01.378 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T06:48:01.378 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:1 2024-09-12T06:48:01.378 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:48:01.378 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:48:01.378 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:48:01.378 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:48:01.378 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T06:48:01.378 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:48:01.378 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:48:01.378 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=1 2024-09-12T06:48:01.378 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:48:01.379 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T06:48:01.391 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:48:01.391 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:48:01.391 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:48:01.391 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_16 2024-09-12T06:48:01.491 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:48:01.491 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror image_16 2024-09-12T06:48:01.491 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:48:01.491 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:48:01.491 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_16 2024-09-12T06:48:01.491 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_16 '' 2024-09-12T06:48:01.491 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:48:01.492 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:48:01.492 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_16 2024-09-12T06:48:01.492 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:48:01.492 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_16.status 2024-09-12T06:48:01.492 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_16.status 2024-09-12T06:48:01.492 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:48:01.493 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_16.status 2024-09-12T06:48:01.493 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_16 2024-09-12T06:48:01.493 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_16.status 2024-09-12T06:48:01.539 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 3 2024-09-12T06:48:01.539 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 3 2024-09-12T06:48:01.539 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:48:01.539 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=11194], [object_number=15, tag_tid=1, entry_tid=11191]]], state=connected] 2024-09-12T06:48:01.540 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=11194], [object_number=15, tag_tid=1, entry_tid=11191]]], state=connected] 2024-09-12T06:48:01.542 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_16.status 2024-09-12T06:48:01.544 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-09-12T06:48:01.545 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror image_16 2024-09-12T06:48:01.545 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:48:01.545 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:48:01.545 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_16 2024-09-12T06:48:01.545 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_16 '..*' 2024-09-12T06:48:01.545 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:48:01.546 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:48:01.546 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_16 2024-09-12T06:48:01.546 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:48:01.546 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_16.status 2024-09-12T06:48:01.546 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_16.status 2024-09-12T06:48:01.546 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:48:01.547 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_16.status 2024-09-12T06:48:01.547 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_16 2024-09-12T06:48:01.547 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_16.status 2024-09-12T06:48:01.592 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 3 2024-09-12T06:48:01.592 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 3 2024-09-12T06:48:01.592 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:48:01.592 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=11194], [object_number=15, tag_tid=1, entry_tid=11191]]], state=connected] 2024-09-12T06:48:01.592 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=11194], [object_number=15, tag_tid=1, entry_tid=11191]]], state=connected] 2024-09-12T06:48:01.595 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_16.status 2024-09-12T06:48:01.597 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-09-12T06:48:01.597 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=1' 2024-09-12T06:48:01.597 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:48:01.597 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_snap_present cluster1 mirror image_16 snap 2024-09-12T06:48:01.597 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:48:01.597 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:01.597 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_16 2024-09-12T06:48:01.597 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:48:01.597 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:48:01.597 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T06:48:01.597 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:48:02.597 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 info mirror/image_16@snap 2024-09-12T06:48:02.643 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:rbd image 'image_16': 2024-09-12T06:48:02.644 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: size 128 MiB in 32 objects 2024-09-12T06:48:02.644 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: order 22 (4 MiB objects) 2024-09-12T06:48:02.644 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: snapshot_count: 1 2024-09-12T06:48:02.644 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: id: 10af915232ee 2024-09-12T06:48:02.644 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: block_name_prefix: rbd_data.10af915232ee 2024-09-12T06:48:02.644 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: format: 2 2024-09-12T06:48:02.644 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: features: layering, exclusive-lock, journaling 2024-09-12T06:48:02.644 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: op_features: 2024-09-12T06:48:02.644 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: flags: 2024-09-12T06:48:02.644 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: create_timestamp: Thu Sep 12 06:40:51 2024 2024-09-12T06:48:02.644 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: access_timestamp: Thu Sep 12 06:40:51 2024 2024-09-12T06:48:02.645 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: modify_timestamp: Thu Sep 12 06:41:52 2024 2024-09-12T06:48:02.645 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: protected: False 2024-09-12T06:48:02.645 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: journal: 10af915232ee 2024-09-12T06:48:02.645 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring state: enabled 2024-09-12T06:48:02.645 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring mode: journal 2024-09-12T06:48:02.645 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring global id: 9ee38269-b168-47e9-baba-f8abc1c31d33 2024-09-12T06:48:02.645 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring primary: false 2024-09-12T06:48:02.647 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:48:02.647 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror image_16 snap 2024-09-12T06:48:02.647 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:02.647 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_16 2024-09-12T06:48:02.647 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:48:02.648 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:48:02.648 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_16.export 2024-09-12T06:48:02.648 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_16.export 2024-09-12T06:48:02.648 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_16.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_16.export 2024-09-12T06:48:02.648 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export image_16@snap /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_16.export 2024-09-12T06:48:02.806 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:48:02.810 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export image_16@snap /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_16.export 2024-09-12T06:48:03.042 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:48:03.048 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_16.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_16.export 2024-09-12T06:48:03.103 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_16.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_16.export 2024-09-12T06:48:03.135 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:48:03.135 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:48:03.135 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_17 2024-09-12T06:48:03.135 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_snap cluster2 mirror image_17 snap 2024-09-12T06:48:03.135 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:48:03.135 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:03.135 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_17 2024-09-12T06:48:03.135 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:48:03.135 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror snap create image_17@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T06:48:03.807 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_started cluster1 mirror image_17 2024-09-12T06:48:03.807 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:48:03.807 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:03.807 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_17 2024-09-12T06:48:03.807 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_state cluster1 mirror image_17 started 2024-09-12T06:48:03.807 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:48:03.807 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:03.807 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_17 2024-09-12T06:48:03.807 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=started 2024-09-12T06:48:03.808 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:48:03.808 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:48:03.808 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:48:04.808 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_replay_state cluster1 mirror image_17 started 2024-09-12T06:48:04.808 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:48:04.808 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:04.809 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_17 2024-09-12T06:48:04.809 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=started 2024-09-12T06:48:04.809 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local status_result 2024-09-12T06:48:04.809 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=stopped 2024-09-12T06:48:04.809 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_17 2024-09-12T06:48:04.809 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -i state 2024-09-12T06:48:04.809 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_instance=cluster1 2024-09-12T06:48:04.809 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:48:04.809 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:48:04.809 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance=cluster1 2024-09-12T06:48:04.809 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local loop_instance 2024-09-12T06:48:04.809 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:48:04.810 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sleep 0 2024-09-12T06:48:04.812 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:48:04.812 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ seq 0 1 2024-09-12T06:48:04.812 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:48:04.812 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_17 2024-09-12T06:48:04.812 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:0 2024-09-12T06:48:04.812 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:48:04.812 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:48:04.812 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:48:04.812 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:48:04.812 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:48:04.812 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:48:04.812 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:48:04.813 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:48:04.813 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:48:04.813 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:48:04.813 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:48:04.813 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:48:04.813 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:48:04.813 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:48:04.813 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:0 2024-09-12T06:48:04.813 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:48:04.813 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:48:04.813 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:48:04.813 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:0 2024-09-12T06:48:04.815 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:48:04.815 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:48:04.815 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:48:04.815 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:48:04.815 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T06:48:04.815 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:48:04.815 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:48:04.815 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=0 2024-09-12T06:48:04.815 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=0 2024-09-12T06:48:04.815 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:48:04.842 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:48:04.843 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:48:04.843 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:48:04.843 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_17 2024-09-12T06:48:04.947 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ return 0 2024-09-12T06:48:04.948 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T06:48:04.948 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo ' "state": "Replaying"' 2024-09-12T06:48:04.948 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep -i Replaying 2024-09-12T06:48:04.949 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: "state": "Replaying" 2024-09-12T06:48:04.951 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_state=started 2024-09-12T06:48:04.951 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test started = started 2024-09-12T06:48:04.951 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:48:04.951 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_17 2024-09-12T06:48:04.951 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:48:04.951 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:48:04.951 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:48:04.952 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:48:04.952 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_17 2024-09-12T06:48:04.952 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' journal = journal ']' 2024-09-12T06:48:04.952 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror image_17 2024-09-12T06:48:04.952 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:48:04.952 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:48:04.952 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:48:04.952 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:48:04.952 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_17 2024-09-12T06:48:04.952 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-12T06:48:04.952 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-12T06:48:04.952 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:48:04.952 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:48:04.953 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:48:05.152 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror image_17 2024-09-12T06:48:05.152 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:48:05.152 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:05.152 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_17 2024-09-12T06:48:05.152 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:48:05.152 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n image_17 ']' 2024-09-12T06:48:05.152 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/image_17' 2024-09-12T06:48:05.152 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_17 2024-09-12T06:48:05.152 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:48:05.152 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:48:05.153 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:48:05.153 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:48:05.153 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:48:05.153 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:48:05.153 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:48:05.153 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:48:05.153 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:48:05.156 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:48:05.156 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_17 2024-09-12T06:48:05.156 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:48:05.156 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:48:05.156 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:48:05.156 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:48:05.156 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:48:05.156 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:48:05.156 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:48:05.156 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:48:05.157 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:48:05.157 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:48:05.157 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:48:05.157 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:48:05.157 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:48:05.157 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:48:05.157 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:48:05.157 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:48:05.157 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:48:05.158 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:48:05.158 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:48:05.158 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:48:05.158 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:48:05.158 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:48:05.158 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:48:05.158 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:48:05.158 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:48:05.158 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:48:05.158 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:48:05.158 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:48:05.158 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:48:05.158 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:48:05.173 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:48:05.173 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:48:05.173 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:48:05.173 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_17 2024-09-12T06:48:05.269 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:48:05.270 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror image_17 2024-09-12T06:48:05.270 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:48:05.271 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:48:05.271 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_17 2024-09-12T06:48:05.271 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_17 '' 2024-09-12T06:48:05.271 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:48:05.271 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:48:05.271 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_17 2024-09-12T06:48:05.271 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:48:05.271 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_17.status 2024-09-12T06:48:05.271 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_17.status 2024-09-12T06:48:05.271 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:48:05.272 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_17.status 2024-09-12T06:48:05.272 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_17 2024-09-12T06:48:05.272 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_17.status 2024-09-12T06:48:05.317 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 1 2024-09-12T06:48:05.317 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 1 2024-09-12T06:48:05.317 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:48:05.317 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=4598], [object_number=7, tag_tid=1, entry_tid=4595]]], state=connected] 2024-09-12T06:48:05.317 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=4598], [object_number=7, tag_tid=1, entry_tid=4595]]], state=connected] 2024-09-12T06:48:05.320 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_17.status 2024-09-12T06:48:05.322 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-12T06:48:05.323 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror image_17 2024-09-12T06:48:05.323 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:48:05.323 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:48:05.323 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_17 2024-09-12T06:48:05.324 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_17 '..*' 2024-09-12T06:48:05.324 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:48:05.324 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:48:05.324 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_17 2024-09-12T06:48:05.324 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:48:05.324 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_17.status 2024-09-12T06:48:05.325 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_17.status 2024-09-12T06:48:05.325 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:48:05.326 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_17.status 2024-09-12T06:48:05.326 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_17 2024-09-12T06:48:05.326 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_17.status 2024-09-12T06:48:05.369 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 1 2024-09-12T06:48:05.369 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 1 2024-09-12T06:48:05.369 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:48:05.369 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=4598], [object_number=7, tag_tid=1, entry_tid=4595]]], state=connected] 2024-09-12T06:48:05.369 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=4598], [object_number=7, tag_tid=1, entry_tid=4595]]], state=connected] 2024-09-12T06:48:05.371 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_17.status 2024-09-12T06:48:05.374 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-12T06:48:05.374 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-09-12T06:48:05.374 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:48:05.374 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_snap_present cluster1 mirror image_17 snap 2024-09-12T06:48:05.374 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:48:05.374 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:05.374 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_17 2024-09-12T06:48:05.374 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:48:05.375 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:48:05.375 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T06:48:05.375 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:48:06.374 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 info mirror/image_17@snap 2024-09-12T06:48:06.424 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:rbd image 'image_17': 2024-09-12T06:48:06.424 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: size 128 MiB in 32 objects 2024-09-12T06:48:06.424 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: order 22 (4 MiB objects) 2024-09-12T06:48:06.424 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: snapshot_count: 1 2024-09-12T06:48:06.424 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: id: 10a8bd1997e8 2024-09-12T06:48:06.424 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: block_name_prefix: rbd_data.10a8bd1997e8 2024-09-12T06:48:06.424 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: format: 2 2024-09-12T06:48:06.424 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: features: layering, exclusive-lock, journaling 2024-09-12T06:48:06.424 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: op_features: 2024-09-12T06:48:06.425 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: flags: 2024-09-12T06:48:06.425 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: create_timestamp: Thu Sep 12 06:41:03 2024 2024-09-12T06:48:06.425 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: access_timestamp: Thu Sep 12 06:41:03 2024 2024-09-12T06:48:06.425 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: modify_timestamp: Thu Sep 12 06:42:04 2024 2024-09-12T06:48:06.425 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: protected: False 2024-09-12T06:48:06.425 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: journal: 10a8bd1997e8 2024-09-12T06:48:06.425 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring state: enabled 2024-09-12T06:48:06.425 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring mode: journal 2024-09-12T06:48:06.425 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring global id: 8e62cc41-741b-442f-bfe5-bfda23b92a11 2024-09-12T06:48:06.425 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring primary: false 2024-09-12T06:48:06.427 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:48:06.428 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror image_17 snap 2024-09-12T06:48:06.428 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:06.428 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_17 2024-09-12T06:48:06.428 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:48:06.428 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:48:06.428 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_17.export 2024-09-12T06:48:06.428 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_17.export 2024-09-12T06:48:06.428 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_17.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_17.export 2024-09-12T06:48:06.428 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export image_17@snap /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_17.export 2024-09-12T06:48:06.576 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:48:06.583 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export image_17@snap /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_17.export 2024-09-12T06:48:06.760 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:48:06.766 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_17.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_17.export 2024-09-12T06:48:06.822 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_17.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_17.export 2024-09-12T06:48:06.853 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:48:06.853 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:48:06.853 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_18 2024-09-12T06:48:06.854 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_snap cluster2 mirror image_18 snap 2024-09-12T06:48:06.854 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:48:06.854 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:06.854 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_18 2024-09-12T06:48:06.854 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:48:06.854 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror snap create image_18@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T06:48:07.823 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_started cluster1 mirror image_18 2024-09-12T06:48:07.823 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:48:07.823 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:07.823 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_18 2024-09-12T06:48:07.823 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_state cluster1 mirror image_18 started 2024-09-12T06:48:07.823 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:48:07.823 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:07.824 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_18 2024-09-12T06:48:07.824 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=started 2024-09-12T06:48:07.824 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:48:07.824 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:48:07.824 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:48:08.825 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_replay_state cluster1 mirror image_18 started 2024-09-12T06:48:08.825 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:48:08.825 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:08.825 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_18 2024-09-12T06:48:08.825 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=started 2024-09-12T06:48:08.825 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local status_result 2024-09-12T06:48:08.826 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=stopped 2024-09-12T06:48:08.826 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_18 2024-09-12T06:48:08.826 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -i state 2024-09-12T06:48:08.826 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_instance=cluster1 2024-09-12T06:48:08.826 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:48:08.826 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:48:08.826 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance=cluster1 2024-09-12T06:48:08.826 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local loop_instance 2024-09-12T06:48:08.826 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:48:08.826 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sleep 0 2024-09-12T06:48:08.828 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:48:08.828 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ seq 0 1 2024-09-12T06:48:08.828 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:48:08.828 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_18 2024-09-12T06:48:08.828 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:0 2024-09-12T06:48:08.828 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:48:08.828 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:48:08.828 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:48:08.828 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:48:08.829 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:48:08.829 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:48:08.829 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:48:08.829 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:48:08.829 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:48:08.829 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:48:08.829 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:48:08.829 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:48:08.829 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:48:08.831 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:48:08.831 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:0 2024-09-12T06:48:08.831 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:48:08.831 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:48:08.831 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:48:08.831 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:0 2024-09-12T06:48:08.831 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:48:08.831 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:48:08.831 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:48:08.832 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:48:08.832 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T06:48:08.832 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:48:08.832 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:48:08.832 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=0 2024-09-12T06:48:08.832 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=0 2024-09-12T06:48:08.832 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:48:08.849 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:48:08.849 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:48:08.849 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:48:08.849 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_18 2024-09-12T06:48:08.947 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2024-09-12T06:48:08.948 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T06:48:08.948 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:admin_socket: invalid command 2024-09-12T06:48:08.951 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:48:08.951 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_18 2024-09-12T06:48:08.951 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:1 2024-09-12T06:48:08.951 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:48:08.951 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:48:08.952 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T06:48:08.952 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:1 2024-09-12T06:48:08.952 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:48:08.952 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:48:08.952 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:48:08.952 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:48:08.952 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T06:48:08.952 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:48:08.952 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:48:08.952 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=1 2024-09-12T06:48:08.952 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:48:08.952 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T06:48:08.952 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:1 2024-09-12T06:48:08.953 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:48:08.953 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:48:08.953 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T06:48:08.953 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:1 2024-09-12T06:48:08.953 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:48:08.953 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:48:08.953 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:48:08.953 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:48:08.953 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T06:48:08.953 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:48:08.953 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:48:08.953 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=1 2024-09-12T06:48:08.953 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=1 2024-09-12T06:48:08.953 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T06:48:08.970 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:48:08.970 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:48:08.970 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:48:08.970 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_18 2024-09-12T06:48:09.072 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ return 0 2024-09-12T06:48:09.073 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T06:48:09.074 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo ' "state": "Replaying"' 2024-09-12T06:48:09.074 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep -i Replaying 2024-09-12T06:48:09.074 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: "state": "Replaying" 2024-09-12T06:48:09.077 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_state=started 2024-09-12T06:48:09.077 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test started = started 2024-09-12T06:48:09.077 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:48:09.078 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_18 2024-09-12T06:48:09.078 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:48:09.078 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:48:09.078 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:48:09.078 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:48:09.078 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_18 2024-09-12T06:48:09.078 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' journal = journal ']' 2024-09-12T06:48:09.078 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror image_18 2024-09-12T06:48:09.078 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:48:09.078 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:48:09.078 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:48:09.078 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:48:09.078 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_18 2024-09-12T06:48:09.078 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-12T06:48:09.079 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-12T06:48:09.079 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:48:09.079 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:48:09.079 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:48:09.277 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror image_18 2024-09-12T06:48:09.277 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:48:09.277 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:09.277 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_18 2024-09-12T06:48:09.277 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:48:09.277 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n image_18 ']' 2024-09-12T06:48:09.277 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/image_18' 2024-09-12T06:48:09.277 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_18 2024-09-12T06:48:09.278 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:48:09.278 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:48:09.278 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:48:09.278 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:48:09.278 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:48:09.278 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:48:09.278 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:48:09.278 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:48:09.278 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:48:09.281 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:48:09.281 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_18 2024-09-12T06:48:09.281 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:48:09.282 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:48:09.282 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:48:09.282 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:48:09.282 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:48:09.282 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:48:09.282 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:48:09.282 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:48:09.282 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:48:09.282 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:48:09.282 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:48:09.282 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:48:09.282 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:48:09.282 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:48:09.283 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:48:09.283 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:48:09.283 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:48:09.283 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:48:09.283 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:48:09.283 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:48:09.283 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:48:09.283 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:48:09.283 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:48:09.283 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:48:09.283 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:48:09.283 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:48:09.283 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:48:09.283 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:48:09.284 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:48:09.284 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:48:09.299 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:48:09.299 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:48:09.299 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:48:09.299 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_18 2024-09-12T06:48:09.386 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2024-09-12T06:48:09.386 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd mirror flush : flush rbd mirror 2024-09-12T06:48:09.386 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:admin_socket: invalid command 2024-09-12T06:48:09.390 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:48:09.390 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_18 2024-09-12T06:48:09.390 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:1 2024-09-12T06:48:09.390 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:48:09.390 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:48:09.390 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-12T06:48:09.390 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:1 2024-09-12T06:48:09.390 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:48:09.390 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:48:09.390 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:48:09.391 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=1 2024-09-12T06:48:09.391 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-12T06:48:09.391 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:48:09.391 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:48:09.391 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=1 2024-09-12T06:48:09.391 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:48:09.391 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-12T06:48:09.391 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:1 2024-09-12T06:48:09.391 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:48:09.391 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:48:09.391 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T06:48:09.391 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:1 2024-09-12T06:48:09.391 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:48:09.392 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:48:09.392 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:48:09.392 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:48:09.392 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T06:48:09.392 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:48:09.392 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:48:09.392 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=1 2024-09-12T06:48:09.392 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:48:09.392 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T06:48:09.408 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:48:09.408 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:48:09.409 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:48:09.409 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_18 2024-09-12T06:48:09.508 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:48:09.509 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror image_18 2024-09-12T06:48:09.510 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:48:09.510 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:48:09.510 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_18 2024-09-12T06:48:09.510 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_18 '' 2024-09-12T06:48:09.510 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:48:09.510 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:48:09.510 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_18 2024-09-12T06:48:09.510 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:48:09.510 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_18.status 2024-09-12T06:48:09.510 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_18.status 2024-09-12T06:48:09.511 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:48:09.511 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_18.status 2024-09-12T06:48:09.511 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_18 2024-09-12T06:48:09.511 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_18.status 2024-09-12T06:48:09.556 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 1 2024-09-12T06:48:09.557 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 1 2024-09-12T06:48:09.557 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:48:09.557 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4815], [object_number=6, tag_tid=1, entry_tid=4814]]], state=connected] 2024-09-12T06:48:09.557 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4815], [object_number=6, tag_tid=1, entry_tid=4814]]], state=connected] 2024-09-12T06:48:09.559 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_18.status 2024-09-12T06:48:09.561 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-12T06:48:09.562 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror image_18 2024-09-12T06:48:09.562 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:48:09.562 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:48:09.562 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_18 2024-09-12T06:48:09.562 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_18 '..*' 2024-09-12T06:48:09.562 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:48:09.562 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:48:09.563 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_18 2024-09-12T06:48:09.563 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:48:09.563 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_18.status 2024-09-12T06:48:09.563 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_18.status 2024-09-12T06:48:09.563 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:48:09.563 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_18.status 2024-09-12T06:48:09.563 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_18 2024-09-12T06:48:09.564 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_18.status 2024-09-12T06:48:09.609 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 1 2024-09-12T06:48:09.609 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 1 2024-09-12T06:48:09.609 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:48:09.609 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4815], [object_number=6, tag_tid=1, entry_tid=4814]]], state=connected] 2024-09-12T06:48:09.609 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4815], [object_number=6, tag_tid=1, entry_tid=4814]]], state=connected] 2024-09-12T06:48:09.623 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_18.status 2024-09-12T06:48:09.625 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-12T06:48:09.625 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-09-12T06:48:09.625 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:48:09.625 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_snap_present cluster1 mirror image_18 snap 2024-09-12T06:48:09.626 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:48:09.626 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:09.626 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_18 2024-09-12T06:48:09.626 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:48:09.626 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:48:09.626 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T06:48:09.626 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:48:10.626 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 info mirror/image_18@snap 2024-09-12T06:48:10.673 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:rbd image 'image_18': 2024-09-12T06:48:10.673 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: size 128 MiB in 32 objects 2024-09-12T06:48:10.673 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: order 22 (4 MiB objects) 2024-09-12T06:48:10.673 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: snapshot_count: 1 2024-09-12T06:48:10.673 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: id: 10af8c5f335f 2024-09-12T06:48:10.673 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: block_name_prefix: rbd_data.10af8c5f335f 2024-09-12T06:48:10.673 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: format: 2 2024-09-12T06:48:10.673 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: features: layering, exclusive-lock, journaling 2024-09-12T06:48:10.673 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: op_features: 2024-09-12T06:48:10.673 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: flags: 2024-09-12T06:48:10.673 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: create_timestamp: Thu Sep 12 06:41:08 2024 2024-09-12T06:48:10.674 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: access_timestamp: Thu Sep 12 06:41:08 2024 2024-09-12T06:48:10.674 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: modify_timestamp: Thu Sep 12 06:42:09 2024 2024-09-12T06:48:10.674 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: protected: False 2024-09-12T06:48:10.674 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: journal: 10af8c5f335f 2024-09-12T06:48:10.674 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring state: enabled 2024-09-12T06:48:10.674 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring mode: journal 2024-09-12T06:48:10.674 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring global id: c737391b-68e9-44f1-bb1c-ddaa010a26a8 2024-09-12T06:48:10.674 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring primary: false 2024-09-12T06:48:10.676 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:48:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror image_18 snap 2024-09-12T06:48:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_18 2024-09-12T06:48:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:48:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:48:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_18.export 2024-09-12T06:48:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_18.export 2024-09-12T06:48:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_18.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_18.export 2024-09-12T06:48:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export image_18@snap /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_18.export 2024-09-12T06:48:10.820 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:48:10.827 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export image_18@snap /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_18.export 2024-09-12T06:48:10.999 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:48:11.005 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_18.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_18.export 2024-09-12T06:48:11.062 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_18.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_18.export 2024-09-12T06:48:11.093 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:48:11.093 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:48:11.093 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_19 2024-09-12T06:48:11.093 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_snap cluster2 mirror image_19 snap 2024-09-12T06:48:11.093 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:48:11.093 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:11.093 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_19 2024-09-12T06:48:11.093 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:48:11.093 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror snap create image_19@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T06:48:11.832 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_started cluster1 mirror image_19 2024-09-12T06:48:11.832 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:48:11.832 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:11.833 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_19 2024-09-12T06:48:11.833 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_state cluster1 mirror image_19 started 2024-09-12T06:48:11.833 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:48:11.833 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:11.833 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_19 2024-09-12T06:48:11.833 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=started 2024-09-12T06:48:11.833 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:48:11.833 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:48:11.833 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:48:12.834 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_replay_state cluster1 mirror image_19 started 2024-09-12T06:48:12.834 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:48:12.834 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:12.834 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_19 2024-09-12T06:48:12.834 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=started 2024-09-12T06:48:12.834 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local status_result 2024-09-12T06:48:12.835 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=stopped 2024-09-12T06:48:12.835 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_19 2024-09-12T06:48:12.835 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_instance=cluster1 2024-09-12T06:48:12.835 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -i state 2024-09-12T06:48:12.835 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:48:12.835 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:48:12.835 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance=cluster1 2024-09-12T06:48:12.835 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local loop_instance 2024-09-12T06:48:12.835 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:48:12.835 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sleep 0 2024-09-12T06:48:12.838 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:48:12.839 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ seq 0 1 2024-09-12T06:48:12.839 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:48:12.839 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_19 2024-09-12T06:48:12.839 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:0 2024-09-12T06:48:12.839 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:48:12.839 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:48:12.839 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:48:12.839 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:48:12.839 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:48:12.839 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:48:12.839 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:48:12.839 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:48:12.839 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:48:12.840 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:48:12.840 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:48:12.840 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:48:12.840 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:48:12.840 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:48:12.840 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:0 2024-09-12T06:48:12.840 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:48:12.840 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:48:12.840 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:48:12.840 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:0 2024-09-12T06:48:12.840 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:48:12.841 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:48:12.841 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:48:12.841 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:48:12.841 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T06:48:12.841 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:48:12.841 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:48:12.841 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=0 2024-09-12T06:48:12.841 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=0 2024-09-12T06:48:12.841 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:48:12.856 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:48:12.856 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:48:12.856 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:48:12.856 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_19 2024-09-12T06:48:12.954 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ return 0 2024-09-12T06:48:12.954 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T06:48:12.955 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo ' "state": "Replaying"' 2024-09-12T06:48:12.955 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep -i Replaying 2024-09-12T06:48:12.955 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: "state": "Replaying" 2024-09-12T06:48:12.957 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_state=started 2024-09-12T06:48:12.957 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test started = started 2024-09-12T06:48:12.957 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:48:12.957 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_19 2024-09-12T06:48:12.957 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:48:12.957 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:48:12.957 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:48:12.958 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:48:12.958 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_19 2024-09-12T06:48:12.958 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' journal = journal ']' 2024-09-12T06:48:12.958 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror image_19 2024-09-12T06:48:12.958 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:48:12.958 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:48:12.959 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:48:12.959 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:48:12.959 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_19 2024-09-12T06:48:12.959 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-12T06:48:12.959 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-12T06:48:12.959 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:48:12.959 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:48:12.959 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:48:13.158 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror image_19 2024-09-12T06:48:13.158 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:48:13.158 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:13.158 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_19 2024-09-12T06:48:13.158 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:48:13.159 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n image_19 ']' 2024-09-12T06:48:13.159 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/image_19' 2024-09-12T06:48:13.159 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_19 2024-09-12T06:48:13.159 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:48:13.159 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:48:13.159 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:48:13.159 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:48:13.159 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:48:13.159 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:48:13.159 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:48:13.159 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:48:13.159 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:48:13.162 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:48:13.162 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_19 2024-09-12T06:48:13.162 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:48:13.162 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:48:13.162 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:48:13.163 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:48:13.163 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:48:13.163 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:48:13.163 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:48:13.163 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:48:13.163 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:48:13.163 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:48:13.163 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:48:13.163 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:48:13.163 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:48:13.163 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:48:13.163 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:48:13.163 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:48:13.163 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:48:13.164 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:48:13.164 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:48:13.164 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:48:13.164 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:48:13.164 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:48:13.164 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:48:13.164 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:48:13.164 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:48:13.164 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:48:13.164 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:48:13.164 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:48:13.164 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:48:13.164 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:48:13.180 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:48:13.180 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:48:13.180 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:48:13.180 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_19 2024-09-12T06:48:13.278 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:48:13.278 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror image_19 2024-09-12T06:48:13.278 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:48:13.278 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:48:13.278 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_19 2024-09-12T06:48:13.279 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_19 '' 2024-09-12T06:48:13.279 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:48:13.279 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:48:13.279 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_19 2024-09-12T06:48:13.279 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:48:13.279 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_19.status 2024-09-12T06:48:13.279 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_19.status 2024-09-12T06:48:13.279 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:48:13.280 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_19.status 2024-09-12T06:48:13.280 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_19 2024-09-12T06:48:13.280 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_19.status 2024-09-12T06:48:13.323 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 1 2024-09-12T06:48:13.323 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 1 2024-09-12T06:48:13.324 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:48:13.324 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4539], [object_number=6, tag_tid=1, entry_tid=4538]]], state=connected] 2024-09-12T06:48:13.324 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4539], [object_number=6, tag_tid=1, entry_tid=4538]]], state=connected] 2024-09-12T06:48:13.326 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_19.status 2024-09-12T06:48:13.328 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-12T06:48:13.329 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror image_19 2024-09-12T06:48:13.329 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:48:13.329 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:48:13.330 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_19 2024-09-12T06:48:13.330 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_19 '..*' 2024-09-12T06:48:13.330 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:48:13.330 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:48:13.330 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_19 2024-09-12T06:48:13.330 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:48:13.330 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_19.status 2024-09-12T06:48:13.330 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_19.status 2024-09-12T06:48:13.330 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:48:13.331 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_19.status 2024-09-12T06:48:13.331 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_19 2024-09-12T06:48:13.331 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_19.status 2024-09-12T06:48:13.376 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 1 2024-09-12T06:48:13.376 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 1 2024-09-12T06:48:13.377 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:48:13.377 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4539], [object_number=6, tag_tid=1, entry_tid=4538]]], state=connected] 2024-09-12T06:48:13.377 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4539], [object_number=6, tag_tid=1, entry_tid=4538]]], state=connected] 2024-09-12T06:48:13.379 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_19.status 2024-09-12T06:48:13.381 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-12T06:48:13.381 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-09-12T06:48:13.381 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:48:13.382 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_snap_present cluster1 mirror image_19 snap 2024-09-12T06:48:13.382 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:48:13.382 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:13.382 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_19 2024-09-12T06:48:13.382 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:48:13.382 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:48:13.382 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T06:48:13.382 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:48:14.382 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 info mirror/image_19@snap 2024-09-12T06:48:14.433 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:rbd image 'image_19': 2024-09-12T06:48:14.433 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: size 128 MiB in 32 objects 2024-09-12T06:48:14.433 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: order 22 (4 MiB objects) 2024-09-12T06:48:14.433 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: snapshot_count: 1 2024-09-12T06:48:14.433 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: id: 10a86082fb66 2024-09-12T06:48:14.433 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: block_name_prefix: rbd_data.10a86082fb66 2024-09-12T06:48:14.433 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: format: 2 2024-09-12T06:48:14.433 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: features: layering, exclusive-lock, journaling 2024-09-12T06:48:14.434 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: op_features: 2024-09-12T06:48:14.434 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: flags: 2024-09-12T06:48:14.434 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: create_timestamp: Thu Sep 12 06:41:14 2024 2024-09-12T06:48:14.434 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: access_timestamp: Thu Sep 12 06:41:14 2024 2024-09-12T06:48:14.434 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: modify_timestamp: Thu Sep 12 06:42:15 2024 2024-09-12T06:48:14.434 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: protected: False 2024-09-12T06:48:14.434 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: journal: 10a86082fb66 2024-09-12T06:48:14.434 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring state: enabled 2024-09-12T06:48:14.434 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring mode: journal 2024-09-12T06:48:14.434 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring global id: 088d4084-07e7-4dc2-8885-e4e76cb5f05c 2024-09-12T06:48:14.434 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring primary: false 2024-09-12T06:48:14.437 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:48:14.437 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror image_19 snap 2024-09-12T06:48:14.437 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:14.437 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_19 2024-09-12T06:48:14.437 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:48:14.437 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:48:14.437 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_19.export 2024-09-12T06:48:14.437 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_19.export 2024-09-12T06:48:14.437 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_19.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_19.export 2024-09-12T06:48:14.438 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export image_19@snap /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_19.export 2024-09-12T06:48:14.585 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:48:14.591 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export image_19@snap /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_19.export 2024-09-12T06:48:14.763 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:48:14.768 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_19.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_19.export 2024-09-12T06:48:14.839 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_19.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_19.export 2024-09-12T06:48:14.871 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:48:14.871 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:48:14.871 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_20 2024-09-12T06:48:14.871 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_snap cluster2 mirror image_20 snap 2024-09-12T06:48:14.871 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:48:14.871 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:14.871 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_20 2024-09-12T06:48:14.871 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:48:14.871 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror snap create image_20@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T06:48:17.864 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_started cluster1 mirror image_20 2024-09-12T06:48:17.864 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:48:17.864 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:17.864 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_20 2024-09-12T06:48:17.864 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_state cluster1 mirror image_20 started 2024-09-12T06:48:17.864 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:48:17.864 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:17.864 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_20 2024-09-12T06:48:17.865 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=started 2024-09-12T06:48:17.865 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:48:17.865 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:48:17.865 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:48:18.865 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_replay_state cluster1 mirror image_20 started 2024-09-12T06:48:18.866 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:48:18.866 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:18.866 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_20 2024-09-12T06:48:18.866 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=started 2024-09-12T06:48:18.866 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local status_result 2024-09-12T06:48:18.866 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=stopped 2024-09-12T06:48:18.866 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_20 2024-09-12T06:48:18.866 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_instance=cluster1 2024-09-12T06:48:18.866 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:48:18.866 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -i state 2024-09-12T06:48:18.866 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:48:18.866 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance=cluster1 2024-09-12T06:48:18.867 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local loop_instance 2024-09-12T06:48:18.867 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:48:18.867 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sleep 0 2024-09-12T06:48:18.870 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:48:18.870 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ seq 0 1 2024-09-12T06:48:18.870 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:48:18.870 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_20 2024-09-12T06:48:18.870 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:0 2024-09-12T06:48:18.870 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:48:18.870 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:48:18.871 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:48:18.871 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:48:18.871 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:48:18.871 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:48:18.871 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:48:18.871 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:48:18.871 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:48:18.871 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:48:18.871 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:48:18.871 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:48:18.871 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:48:18.871 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:48:18.871 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:0 2024-09-12T06:48:18.871 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:48:18.871 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:48:18.872 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:48:18.872 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:0 2024-09-12T06:48:18.872 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:48:18.872 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:48:18.872 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:48:18.872 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:48:18.872 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T06:48:18.872 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:48:18.872 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:48:18.872 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=0 2024-09-12T06:48:18.872 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=0 2024-09-12T06:48:18.872 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:48:18.888 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:48:18.888 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:48:18.888 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:48:18.888 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_20 2024-09-12T06:48:18.981 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2024-09-12T06:48:18.981 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T06:48:18.981 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:admin_socket: invalid command 2024-09-12T06:48:18.985 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:48:18.985 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_20 2024-09-12T06:48:18.985 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:1 2024-09-12T06:48:18.985 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:48:18.986 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:48:18.986 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T06:48:18.986 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:1 2024-09-12T06:48:18.986 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:48:18.986 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:48:18.986 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:48:18.986 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:48:18.986 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T06:48:18.986 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:48:18.986 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:48:18.986 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=1 2024-09-12T06:48:18.986 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:48:18.986 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T06:48:18.986 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:1 2024-09-12T06:48:18.987 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:48:18.987 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:48:18.987 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T06:48:18.987 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:1 2024-09-12T06:48:18.987 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:48:18.987 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:48:18.987 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:48:18.987 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:48:18.987 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T06:48:18.987 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:48:18.987 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:48:18.987 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=1 2024-09-12T06:48:18.987 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=1 2024-09-12T06:48:18.987 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T06:48:19.003 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:48:19.004 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:48:19.004 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:48:19.004 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_20 2024-09-12T06:48:19.103 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ return 0 2024-09-12T06:48:19.104 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T06:48:19.104 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo ' "state": "Replaying"' 2024-09-12T06:48:19.104 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep -i Replaying 2024-09-12T06:48:19.105 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: "state": "Replaying" 2024-09-12T06:48:19.107 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_state=started 2024-09-12T06:48:19.107 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test started = started 2024-09-12T06:48:19.107 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:48:19.107 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_20 2024-09-12T06:48:19.107 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:48:19.107 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:48:19.107 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:48:19.107 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:48:19.107 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_20 2024-09-12T06:48:19.107 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' journal = journal ']' 2024-09-12T06:48:19.108 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror image_20 2024-09-12T06:48:19.108 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:48:19.108 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:48:19.108 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:48:19.108 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:48:19.108 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_20 2024-09-12T06:48:19.108 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-12T06:48:19.108 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-12T06:48:19.108 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:48:19.108 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:48:19.108 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:48:19.308 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror image_20 2024-09-12T06:48:19.308 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:48:19.308 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:19.308 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_20 2024-09-12T06:48:19.308 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:48:19.308 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n image_20 ']' 2024-09-12T06:48:19.308 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/image_20' 2024-09-12T06:48:19.308 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_20 2024-09-12T06:48:19.308 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:48:19.308 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:48:19.308 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:48:19.309 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:48:19.309 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:48:19.309 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:48:19.309 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:48:19.309 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:48:19.309 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:48:19.312 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:48:19.312 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_20 2024-09-12T06:48:19.312 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:48:19.312 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:48:19.312 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:48:19.312 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:48:19.312 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:48:19.312 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:48:19.312 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:48:19.313 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:48:19.313 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:48:19.313 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:48:19.313 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:48:19.313 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:48:19.313 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:48:19.313 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:48:19.313 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:48:19.313 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:48:19.313 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:48:19.313 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:48:19.313 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:48:19.313 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:48:19.314 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:48:19.314 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:48:19.314 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:48:19.314 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:48:19.314 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:48:19.314 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:48:19.314 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:48:19.314 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:48:19.314 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:48:19.314 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:48:19.329 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:48:19.329 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:48:19.329 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:48:19.329 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_20 2024-09-12T06:48:19.415 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2024-09-12T06:48:19.415 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd mirror flush : flush rbd mirror 2024-09-12T06:48:19.415 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:admin_socket: invalid command 2024-09-12T06:48:19.418 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:48:19.419 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_20 2024-09-12T06:48:19.419 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:1 2024-09-12T06:48:19.419 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:48:19.419 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:48:19.419 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-12T06:48:19.419 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:1 2024-09-12T06:48:19.419 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:48:19.419 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:48:19.419 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:48:19.419 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=1 2024-09-12T06:48:19.419 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-12T06:48:19.420 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:48:19.420 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:48:19.420 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=1 2024-09-12T06:48:19.420 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:48:19.420 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-12T06:48:19.420 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:1 2024-09-12T06:48:19.420 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:48:19.420 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:48:19.420 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T06:48:19.420 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:1 2024-09-12T06:48:19.420 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:48:19.420 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:48:19.420 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:48:19.420 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:48:19.421 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T06:48:19.421 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:48:19.421 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:48:19.421 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=1 2024-09-12T06:48:19.421 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:48:19.421 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T06:48:19.437 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:48:19.437 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:48:19.437 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:48:19.437 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_20 2024-09-12T06:48:19.530 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:48:19.530 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror image_20 2024-09-12T06:48:19.530 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:48:19.530 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:48:19.531 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_20 2024-09-12T06:48:19.531 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_20 '' 2024-09-12T06:48:19.531 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:48:19.531 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:48:19.531 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_20 2024-09-12T06:48:19.532 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:48:19.532 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_20.status 2024-09-12T06:48:19.532 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_20.status 2024-09-12T06:48:19.532 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:48:19.532 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_20.status 2024-09-12T06:48:19.532 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_20 2024-09-12T06:48:19.532 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_20.status 2024-09-12T06:48:19.578 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 3 2024-09-12T06:48:19.578 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 3 2024-09-12T06:48:19.578 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:48:19.578 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=12327], [object_number=14, tag_tid=1, entry_tid=12326]]], state=connected] 2024-09-12T06:48:19.578 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=13, tag_tid=1, entry_tid=12329], [object_number=12, tag_tid=1, entry_tid=12328], [object_number=15, tag_tid=1, entry_tid=12327], [object_number=14, tag_tid=1, entry_tid=12326]]], state=connected] 2024-09-12T06:48:19.580 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_20.status 2024-09-12T06:48:19.583 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-09-12T06:48:19.583 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror image_20 2024-09-12T06:48:19.583 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:48:19.583 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:48:19.583 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_20 2024-09-12T06:48:19.583 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_20 '..*' 2024-09-12T06:48:19.583 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:48:19.583 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:48:19.583 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_20 2024-09-12T06:48:19.584 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:48:19.584 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_20.status 2024-09-12T06:48:19.584 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_20.status 2024-09-12T06:48:19.584 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:48:19.584 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_20.status 2024-09-12T06:48:19.584 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_20 2024-09-12T06:48:19.585 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_20.status 2024-09-12T06:48:19.630 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 3 2024-09-12T06:48:19.631 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 3 2024-09-12T06:48:19.631 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:48:19.631 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=12327], [object_number=14, tag_tid=1, entry_tid=12326]]], state=connected] 2024-09-12T06:48:19.631 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=13, tag_tid=1, entry_tid=12329], [object_number=12, tag_tid=1, entry_tid=12328], [object_number=15, tag_tid=1, entry_tid=12327], [object_number=14, tag_tid=1, entry_tid=12326]]], state=connected] 2024-09-12T06:48:19.633 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_20.status 2024-09-12T06:48:19.635 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=13, tag_tid=1, entry_tid=12329' 2024-09-12T06:48:19.635 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=13, tag_tid=1, entry_tid=12329' 2024-09-12T06:48:19.635 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 'object_number=13, tag_tid=1, entry_tid=12329' '!=' '' 2024-09-12T06:48:19.635 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ break 2024-09-12T06:48:19.635 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 'object_number=13, tag_tid=1, entry_tid=12329' = '' 2024-09-12T06:48:19.635 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ last_mirror_pos='object_number=13, tag_tid=1, entry_tid=12329' 2024-09-12T06:48:19.635 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=1' 2024-09-12T06:48:19.636 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-12T06:48:19.636 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:48:19.636 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_tag=2 2024-09-12T06:48:19.637 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=13, tag_tid=1, entry_tid=12329' 2024-09-12T06:48:19.637 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-12T06:48:19.637 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:48:19.638 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_tag=1 2024-09-12T06:48:19.638 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=1' 2024-09-12T06:48:19.638 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-12T06:48:19.639 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:48:19.640 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_entry=1 2024-09-12T06:48:19.640 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=13, tag_tid=1, entry_tid=12329' 2024-09-12T06:48:19.640 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-12T06:48:19.640 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:48:19.641 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_entry=12329 2024-09-12T06:48:19.641 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 2 = 1 -a 1 -le 12329 2024-09-12T06:48:19.641 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:48:19.641 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:48:19.641 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:48:19.843 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror image_20 2024-09-12T06:48:19.844 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:48:19.844 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:19.844 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_20 2024-09-12T06:48:19.844 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:48:19.844 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n image_20 ']' 2024-09-12T06:48:19.844 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/image_20' 2024-09-12T06:48:19.844 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_20 2024-09-12T06:48:19.844 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:48:19.844 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:48:19.844 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:48:19.844 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:48:19.845 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:48:19.845 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:48:19.845 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:48:19.845 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:48:19.845 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:48:19.848 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:48:19.848 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_20 2024-09-12T06:48:19.848 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:48:19.848 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:48:19.848 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:48:19.848 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:48:19.848 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:48:19.848 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:48:19.849 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:48:19.849 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:48:19.849 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:48:19.849 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:48:19.849 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:48:19.849 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:48:19.849 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:48:19.849 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:48:19.849 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:48:19.849 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:48:19.849 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:48:19.849 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:48:19.849 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:48:19.849 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:48:19.850 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:48:19.850 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:48:19.850 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:48:19.850 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:48:19.850 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:48:19.850 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:48:19.850 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:48:19.850 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:48:19.850 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:48:19.850 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:48:19.865 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:48:19.865 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:48:19.865 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:48:19.865 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_20 2024-09-12T06:48:19.954 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2024-09-12T06:48:19.954 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd mirror flush : flush rbd mirror 2024-09-12T06:48:19.954 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:admin_socket: invalid command 2024-09-12T06:48:19.958 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:48:19.958 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_20 2024-09-12T06:48:19.958 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:1 2024-09-12T06:48:19.958 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:48:19.958 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:48:19.958 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-12T06:48:19.959 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:1 2024-09-12T06:48:19.959 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:48:19.959 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:48:19.959 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:48:19.959 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=1 2024-09-12T06:48:19.959 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-12T06:48:19.959 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:48:19.959 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:48:19.959 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=1 2024-09-12T06:48:19.959 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:48:19.959 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-12T06:48:19.959 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:1 2024-09-12T06:48:19.959 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:48:19.959 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:48:19.960 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T06:48:19.960 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:1 2024-09-12T06:48:19.960 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:48:19.960 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:48:19.960 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:48:19.960 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:48:19.960 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T06:48:19.960 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:48:19.960 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:48:19.960 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=1 2024-09-12T06:48:19.960 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:48:19.960 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T06:48:19.976 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:48:19.976 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:48:19.976 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:48:19.976 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_20 2024-09-12T06:48:20.078 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:48:20.078 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror image_20 2024-09-12T06:48:20.078 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:48:20.078 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:48:20.078 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_20 2024-09-12T06:48:20.079 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_20 '' 2024-09-12T06:48:20.079 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:48:20.079 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:48:20.079 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_20 2024-09-12T06:48:20.079 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:48:20.079 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_20.status 2024-09-12T06:48:20.079 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_20.status 2024-09-12T06:48:20.079 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:48:20.080 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_20.status 2024-09-12T06:48:20.080 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_20 2024-09-12T06:48:20.080 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_20.status 2024-09-12T06:48:20.125 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 3 2024-09-12T06:48:20.125 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 3 2024-09-12T06:48:20.125 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:48:20.125 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=12327], [object_number=14, tag_tid=1, entry_tid=12326]]], state=connected] 2024-09-12T06:48:20.125 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=12327], [object_number=14, tag_tid=1, entry_tid=12326]]], state=connected] 2024-09-12T06:48:20.128 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_20.status 2024-09-12T06:48:20.129 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-09-12T06:48:20.130 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror image_20 2024-09-12T06:48:20.130 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:48:20.130 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:48:20.130 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_20 2024-09-12T06:48:20.131 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_20 '..*' 2024-09-12T06:48:20.131 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:48:20.131 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:48:20.131 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_20 2024-09-12T06:48:20.131 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:48:20.131 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_20.status 2024-09-12T06:48:20.131 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_20.status 2024-09-12T06:48:20.131 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:48:20.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_20.status 2024-09-12T06:48:20.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_20 2024-09-12T06:48:20.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_20.status 2024-09-12T06:48:20.176 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 3 2024-09-12T06:48:20.177 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 3 2024-09-12T06:48:20.177 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:48:20.177 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=12327], [object_number=14, tag_tid=1, entry_tid=12326]]], state=connected] 2024-09-12T06:48:20.177 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=12327], [object_number=14, tag_tid=1, entry_tid=12326]]], state=connected] 2024-09-12T06:48:20.179 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_20.status 2024-09-12T06:48:20.182 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-09-12T06:48:20.182 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=1' 2024-09-12T06:48:20.182 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:48:20.182 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_snap_present cluster1 mirror image_20 snap 2024-09-12T06:48:20.182 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:48:20.182 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:20.182 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_20 2024-09-12T06:48:20.182 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:48:20.182 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:48:20.182 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T06:48:20.182 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:48:21.182 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 info mirror/image_20@snap 2024-09-12T06:48:21.229 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:rbd image 'image_20': 2024-09-12T06:48:21.229 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: size 128 MiB in 32 objects 2024-09-12T06:48:21.229 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: order 22 (4 MiB objects) 2024-09-12T06:48:21.229 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: snapshot_count: 1 2024-09-12T06:48:21.229 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: id: 10af7c76b991 2024-09-12T06:48:21.230 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: block_name_prefix: rbd_data.10af7c76b991 2024-09-12T06:48:21.230 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: format: 2 2024-09-12T06:48:21.230 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: features: layering, exclusive-lock, journaling 2024-09-12T06:48:21.230 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: op_features: 2024-09-12T06:48:21.230 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: flags: 2024-09-12T06:48:21.230 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: create_timestamp: Thu Sep 12 06:41:18 2024 2024-09-12T06:48:21.230 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: access_timestamp: Thu Sep 12 06:41:18 2024 2024-09-12T06:48:21.230 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: modify_timestamp: Thu Sep 12 06:42:19 2024 2024-09-12T06:48:21.230 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: protected: False 2024-09-12T06:48:21.230 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: journal: 10af7c76b991 2024-09-12T06:48:21.230 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring state: enabled 2024-09-12T06:48:21.230 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring mode: journal 2024-09-12T06:48:21.230 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring global id: 243c7222-f2cb-4561-a3b4-f50537c47169 2024-09-12T06:48:21.230 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring primary: false 2024-09-12T06:48:21.233 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:48:21.233 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror image_20 snap 2024-09-12T06:48:21.233 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:21.233 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_20 2024-09-12T06:48:21.233 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:48:21.233 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:48:21.233 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_20.export 2024-09-12T06:48:21.233 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_20.export 2024-09-12T06:48:21.234 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_20.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_20.export 2024-09-12T06:48:21.234 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export image_20@snap /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_20.export 2024-09-12T06:48:21.394 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:48:21.402 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export image_20@snap /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_20.export 2024-09-12T06:48:21.655 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:48:21.662 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_20.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_20.export 2024-09-12T06:48:21.717 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_20.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_20.export 2024-09-12T06:48:21.748 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:48:21.748 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:48:21.748 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_21 2024-09-12T06:48:21.748 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_snap cluster2 mirror image_21 snap 2024-09-12T06:48:21.749 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:48:21.749 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:21.749 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_21 2024-09-12T06:48:21.749 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:48:21.749 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror snap create image_21@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T06:48:24.893 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_started cluster1 mirror image_21 2024-09-12T06:48:24.894 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:48:24.894 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:24.894 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_21 2024-09-12T06:48:24.894 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_state cluster1 mirror image_21 started 2024-09-12T06:48:24.894 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:48:24.894 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:24.894 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_21 2024-09-12T06:48:24.894 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=started 2024-09-12T06:48:24.894 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:48:24.894 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:48:24.894 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:48:25.895 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_replay_state cluster1 mirror image_21 started 2024-09-12T06:48:25.895 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:48:25.895 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:25.895 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_21 2024-09-12T06:48:25.895 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=started 2024-09-12T06:48:25.896 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local status_result 2024-09-12T06:48:25.896 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=stopped 2024-09-12T06:48:25.896 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_21 2024-09-12T06:48:25.896 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_instance=cluster1 2024-09-12T06:48:25.896 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:48:25.896 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:48:25.896 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -i state 2024-09-12T06:48:25.896 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance=cluster1 2024-09-12T06:48:25.896 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local loop_instance 2024-09-12T06:48:25.896 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:48:25.896 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sleep 0 2024-09-12T06:48:25.898 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:48:25.898 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ seq 0 1 2024-09-12T06:48:25.898 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:48:25.898 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_21 2024-09-12T06:48:25.898 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:0 2024-09-12T06:48:25.898 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:48:25.898 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:48:25.899 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:48:25.899 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:48:25.899 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:48:25.899 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:48:25.899 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:48:25.899 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:48:25.899 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:48:25.899 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:48:25.899 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:48:25.899 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:48:25.899 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:48:25.899 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:48:25.899 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:0 2024-09-12T06:48:25.899 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:48:25.899 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:48:25.901 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:48:25.901 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:0 2024-09-12T06:48:25.901 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:48:25.901 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:48:25.901 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:48:25.901 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:48:25.901 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T06:48:25.901 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:48:25.901 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:48:25.902 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=0 2024-09-12T06:48:25.902 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=0 2024-09-12T06:48:25.902 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:48:25.921 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:48:25.922 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:48:25.922 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:48:25.922 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_21 2024-09-12T06:48:26.028 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ return 0 2024-09-12T06:48:26.028 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T06:48:26.028 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo ' "state": "Replaying"' 2024-09-12T06:48:26.029 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep -i Replaying 2024-09-12T06:48:26.029 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: "state": "Replaying" 2024-09-12T06:48:26.031 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_state=started 2024-09-12T06:48:26.031 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test started = started 2024-09-12T06:48:26.031 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:48:26.031 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_21 2024-09-12T06:48:26.031 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:48:26.031 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:48:26.031 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:48:26.031 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:48:26.031 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_21 2024-09-12T06:48:26.031 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' journal = journal ']' 2024-09-12T06:48:26.031 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror image_21 2024-09-12T06:48:26.032 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:48:26.032 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:48:26.032 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:48:26.032 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:48:26.032 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_21 2024-09-12T06:48:26.032 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-12T06:48:26.032 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-12T06:48:26.032 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:48:26.032 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:48:26.032 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:48:26.231 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror image_21 2024-09-12T06:48:26.232 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:48:26.232 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:26.232 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_21 2024-09-12T06:48:26.232 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:48:26.232 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n image_21 ']' 2024-09-12T06:48:26.232 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/image_21' 2024-09-12T06:48:26.232 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_21 2024-09-12T06:48:26.232 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:48:26.232 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:48:26.232 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:48:26.232 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:48:26.232 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:48:26.233 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:48:26.233 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:48:26.233 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:48:26.233 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:48:26.236 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:48:26.236 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_21 2024-09-12T06:48:26.236 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:48:26.236 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:48:26.236 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:48:26.236 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:48:26.236 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:48:26.236 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:48:26.236 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:48:26.236 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:48:26.236 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:48:26.237 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:48:26.237 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:48:26.237 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:48:26.237 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:48:26.237 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:48:26.237 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:48:26.237 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:48:26.237 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:48:26.238 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:48:26.238 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:48:26.238 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:48:26.238 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:48:26.238 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:48:26.238 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:48:26.238 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:48:26.238 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:48:26.238 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:48:26.238 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:48:26.238 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:48:26.238 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:48:26.238 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:48:26.253 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:48:26.254 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:48:26.254 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:48:26.254 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_21 2024-09-12T06:48:26.351 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:48:26.351 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror image_21 2024-09-12T06:48:26.351 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:48:26.351 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:48:26.351 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_21 2024-09-12T06:48:26.351 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_21 '' 2024-09-12T06:48:26.352 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:48:26.352 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:48:26.352 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_21 2024-09-12T06:48:26.352 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:48:26.352 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_21.status 2024-09-12T06:48:26.352 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_21.status 2024-09-12T06:48:26.352 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:48:26.353 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_21.status 2024-09-12T06:48:26.353 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_21 2024-09-12T06:48:26.353 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_21.status 2024-09-12T06:48:26.398 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 2 2024-09-12T06:48:26.398 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 3 2024-09-12T06:48:26.398 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:48:26.399 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9987], [object_number=10, tag_tid=1, entry_tid=9986]]], state=connected] 2024-09-12T06:48:26.399 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9987], [object_number=10, tag_tid=1, entry_tid=9986]]], state=connected] 2024-09-12T06:48:26.401 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_21.status 2024-09-12T06:48:26.402 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-09-12T06:48:26.404 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror image_21 2024-09-12T06:48:26.404 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:48:26.404 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:48:26.404 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_21 2024-09-12T06:48:26.404 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_21 '..*' 2024-09-12T06:48:26.404 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:48:26.404 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:48:26.404 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_21 2024-09-12T06:48:26.404 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:48:26.404 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_21.status 2024-09-12T06:48:26.404 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_21.status 2024-09-12T06:48:26.404 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:48:26.405 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_21.status 2024-09-12T06:48:26.405 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_21 2024-09-12T06:48:26.405 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_21.status 2024-09-12T06:48:26.452 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 2 2024-09-12T06:48:26.452 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 3 2024-09-12T06:48:26.452 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:48:26.452 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9987], [object_number=10, tag_tid=1, entry_tid=9986]]], state=connected] 2024-09-12T06:48:26.453 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9987], [object_number=10, tag_tid=1, entry_tid=9986]]], state=connected] 2024-09-12T06:48:26.455 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_21.status 2024-09-12T06:48:26.457 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-09-12T06:48:26.457 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=1' 2024-09-12T06:48:26.458 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:48:26.458 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_snap_present cluster1 mirror image_21 snap 2024-09-12T06:48:26.458 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:48:26.458 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:26.458 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_21 2024-09-12T06:48:26.458 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:48:26.458 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:48:26.458 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T06:48:26.458 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:48:27.458 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 info mirror/image_21@snap 2024-09-12T06:48:27.506 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:rbd image 'image_21': 2024-09-12T06:48:27.506 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: size 128 MiB in 32 objects 2024-09-12T06:48:27.507 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: order 22 (4 MiB objects) 2024-09-12T06:48:27.507 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: snapshot_count: 1 2024-09-12T06:48:27.507 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: id: 10a81b5252e1 2024-09-12T06:48:27.507 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: block_name_prefix: rbd_data.10a81b5252e1 2024-09-12T06:48:27.507 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: format: 2 2024-09-12T06:48:27.507 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: features: layering, exclusive-lock, journaling 2024-09-12T06:48:27.507 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: op_features: 2024-09-12T06:48:27.507 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: flags: 2024-09-12T06:48:27.507 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: create_timestamp: Thu Sep 12 06:41:28 2024 2024-09-12T06:48:27.507 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: access_timestamp: Thu Sep 12 06:41:28 2024 2024-09-12T06:48:27.507 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: modify_timestamp: Thu Sep 12 06:42:29 2024 2024-09-12T06:48:27.507 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: protected: False 2024-09-12T06:48:27.508 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: journal: 10a81b5252e1 2024-09-12T06:48:27.508 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring state: enabled 2024-09-12T06:48:27.508 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring mode: journal 2024-09-12T06:48:27.508 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring global id: b121776e-9588-4600-bc07-6435e1688c59 2024-09-12T06:48:27.508 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring primary: false 2024-09-12T06:48:27.510 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:48:27.510 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror image_21 snap 2024-09-12T06:48:27.510 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:27.511 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_21 2024-09-12T06:48:27.511 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:48:27.511 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:48:27.511 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_21.export 2024-09-12T06:48:27.511 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_21.export 2024-09-12T06:48:27.511 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_21.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_21.export 2024-09-12T06:48:27.511 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export image_21@snap /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_21.export 2024-09-12T06:48:27.666 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:48:27.673 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export image_21@snap /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_21.export 2024-09-12T06:48:27.757 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:48:27.759+0000 7fa314eac640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-12T06:48:27.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:48:27.912 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_21.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_21.export 2024-09-12T06:48:27.967 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_21.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_21.export 2024-09-12T06:48:28.001 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:48:28.001 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:48:28.001 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_22 2024-09-12T06:48:28.001 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_snap cluster2 mirror image_22 snap 2024-09-12T06:48:28.001 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:48:28.001 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:28.001 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_22 2024-09-12T06:48:28.001 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:48:28.001 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror snap create image_22@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T06:48:28.893 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_started cluster1 mirror image_22 2024-09-12T06:48:28.894 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:48:28.894 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:28.894 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_22 2024-09-12T06:48:28.894 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_state cluster1 mirror image_22 started 2024-09-12T06:48:28.894 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:48:28.894 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:28.894 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_22 2024-09-12T06:48:28.895 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=started 2024-09-12T06:48:28.895 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:48:28.895 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:48:28.895 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:48:29.895 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_replay_state cluster1 mirror image_22 started 2024-09-12T06:48:29.895 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:48:29.895 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:29.895 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_22 2024-09-12T06:48:29.895 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=started 2024-09-12T06:48:29.895 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local status_result 2024-09-12T06:48:29.895 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=stopped 2024-09-12T06:48:29.895 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_22 2024-09-12T06:48:29.896 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_instance=cluster1 2024-09-12T06:48:29.896 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -i state 2024-09-12T06:48:29.896 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:48:29.896 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:48:29.896 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance=cluster1 2024-09-12T06:48:29.896 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local loop_instance 2024-09-12T06:48:29.896 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:48:29.896 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sleep 0 2024-09-12T06:48:29.898 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:48:29.898 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ seq 0 1 2024-09-12T06:48:29.898 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:48:29.898 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_22 2024-09-12T06:48:29.898 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:0 2024-09-12T06:48:29.898 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:48:29.898 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:48:29.899 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:48:29.899 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:48:29.899 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:48:29.899 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:48:29.899 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:48:29.899 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:48:29.899 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:48:29.899 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:48:29.899 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:48:29.899 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:48:29.899 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:48:29.899 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:48:29.899 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:0 2024-09-12T06:48:29.899 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:48:29.901 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:48:29.901 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:48:29.901 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:0 2024-09-12T06:48:29.902 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:48:29.902 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:48:29.902 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:48:29.902 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:48:29.902 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T06:48:29.902 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:48:29.902 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:48:29.902 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=0 2024-09-12T06:48:29.902 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=0 2024-09-12T06:48:29.902 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:48:29.917 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:48:29.917 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:48:29.917 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:48:29.917 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_22 2024-09-12T06:48:30.013 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:Invalid command: unused arguments: ['mirror/image_22'] 2024-09-12T06:48:30.013 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T06:48:30.013 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:admin_socket: invalid command 2024-09-12T06:48:30.016 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:48:30.017 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_22 2024-09-12T06:48:30.017 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:1 2024-09-12T06:48:30.017 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:48:30.017 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:48:30.017 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T06:48:30.017 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:1 2024-09-12T06:48:30.017 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:48:30.017 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:48:30.017 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:48:30.017 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:48:30.017 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T06:48:30.017 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:48:30.017 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:48:30.018 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=1 2024-09-12T06:48:30.018 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:48:30.018 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T06:48:30.018 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:1 2024-09-12T06:48:30.018 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:48:30.018 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:48:30.018 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T06:48:30.018 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:1 2024-09-12T06:48:30.018 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:48:30.018 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:48:30.019 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:48:30.019 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:48:30.019 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T06:48:30.019 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:48:30.019 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:48:30.019 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=1 2024-09-12T06:48:30.019 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=1 2024-09-12T06:48:30.019 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T06:48:30.035 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:48:30.035 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:48:30.035 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:48:30.035 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_22 2024-09-12T06:48:30.136 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ return 0 2024-09-12T06:48:30.137 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T06:48:30.137 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo ' "state": "Replaying"' 2024-09-12T06:48:30.137 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep -i Replaying 2024-09-12T06:48:30.138 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: "state": "Replaying" 2024-09-12T06:48:30.140 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_state=started 2024-09-12T06:48:30.141 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test started = started 2024-09-12T06:48:30.141 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:48:30.141 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_22 2024-09-12T06:48:30.141 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:48:30.141 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:48:30.141 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:48:30.141 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:48:30.141 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_22 2024-09-12T06:48:30.141 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' journal = journal ']' 2024-09-12T06:48:30.141 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror image_22 2024-09-12T06:48:30.141 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:48:30.141 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:48:30.142 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:48:30.142 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:48:30.142 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_22 2024-09-12T06:48:30.142 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-12T06:48:30.142 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-12T06:48:30.142 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:48:30.142 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:48:30.142 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:48:30.340 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror image_22 2024-09-12T06:48:30.340 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:48:30.341 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:30.341 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_22 2024-09-12T06:48:30.341 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:48:30.341 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n image_22 ']' 2024-09-12T06:48:30.341 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/image_22' 2024-09-12T06:48:30.341 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_22 2024-09-12T06:48:30.341 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:48:30.341 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:48:30.341 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:48:30.341 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:48:30.341 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:48:30.341 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:48:30.342 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:48:30.342 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:48:30.342 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:48:30.345 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:48:30.345 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_22 2024-09-12T06:48:30.345 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:48:30.345 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:48:30.345 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:48:30.345 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:48:30.345 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:48:30.345 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:48:30.345 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:48:30.345 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:48:30.345 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:48:30.346 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:48:30.346 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:48:30.346 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:48:30.346 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:48:30.346 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:48:30.346 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:48:30.346 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:48:30.346 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:48:30.346 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:48:30.346 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:48:30.346 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:48:30.346 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:48:30.346 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:48:30.346 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:48:30.347 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:48:30.347 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:48:30.347 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:48:30.347 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:48:30.347 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:48:30.347 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:48:30.347 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:48:30.362 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:48:30.362 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:48:30.362 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:48:30.362 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_22 2024-09-12T06:48:30.448 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:Invalid command: unused arguments: ['mirror/image_22'] 2024-09-12T06:48:30.448 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd mirror flush : flush rbd mirror 2024-09-12T06:48:30.449 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:admin_socket: invalid command 2024-09-12T06:48:30.452 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:48:30.452 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_22 2024-09-12T06:48:30.453 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:1 2024-09-12T06:48:30.453 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:48:30.453 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:48:30.453 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-12T06:48:30.453 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:1 2024-09-12T06:48:30.453 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:48:30.453 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:48:30.453 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:48:30.453 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=1 2024-09-12T06:48:30.453 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-12T06:48:30.453 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:48:30.453 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:48:30.453 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=1 2024-09-12T06:48:30.454 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:48:30.454 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-12T06:48:30.454 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:1 2024-09-12T06:48:30.454 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:48:30.454 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:48:30.454 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T06:48:30.454 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:1 2024-09-12T06:48:30.454 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:48:30.454 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:48:30.454 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:48:30.454 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:48:30.454 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T06:48:30.454 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:48:30.455 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:48:30.455 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=1 2024-09-12T06:48:30.455 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:48:30.455 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T06:48:30.471 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:48:30.471 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:48:30.471 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:48:30.471 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_22 2024-09-12T06:48:30.568 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:48:30.568 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror image_22 2024-09-12T06:48:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:48:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:48:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_22 2024-09-12T06:48:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_22 '' 2024-09-12T06:48:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:48:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:48:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_22 2024-09-12T06:48:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:48:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_22.status 2024-09-12T06:48:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_22.status 2024-09-12T06:48:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:48:30.570 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_22.status 2024-09-12T06:48:30.570 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_22 2024-09-12T06:48:30.570 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_22.status 2024-09-12T06:48:30.615 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 1 2024-09-12T06:48:30.616 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 1 2024-09-12T06:48:30.616 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:48:30.616 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=3931], [object_number=6, tag_tid=1, entry_tid=3930]]], state=connected] 2024-09-12T06:48:30.616 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=3931], [object_number=6, tag_tid=1, entry_tid=3930]]], state=connected] 2024-09-12T06:48:30.618 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_22.status 2024-09-12T06:48:30.621 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-12T06:48:30.621 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror image_22 2024-09-12T06:48:30.621 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:48:30.622 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:48:30.622 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_22 2024-09-12T06:48:30.622 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_22 '..*' 2024-09-12T06:48:30.622 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:48:30.622 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:48:30.622 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_22 2024-09-12T06:48:30.622 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:48:30.622 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_22.status 2024-09-12T06:48:30.622 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_22.status 2024-09-12T06:48:30.622 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:48:30.623 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_22.status 2024-09-12T06:48:30.623 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_22 2024-09-12T06:48:30.623 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_22.status 2024-09-12T06:48:30.667 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 1 2024-09-12T06:48:30.667 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 1 2024-09-12T06:48:30.667 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:48:30.667 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=3931], [object_number=6, tag_tid=1, entry_tid=3930]]], state=connected] 2024-09-12T06:48:30.667 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=3931], [object_number=6, tag_tid=1, entry_tid=3930]]], state=connected] 2024-09-12T06:48:30.670 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_22.status 2024-09-12T06:48:30.672 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-12T06:48:30.672 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-09-12T06:48:30.672 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:48:30.673 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_snap_present cluster1 mirror image_22 snap 2024-09-12T06:48:30.673 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:48:30.673 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:30.673 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_22 2024-09-12T06:48:30.673 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:48:30.673 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:48:30.673 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T06:48:30.673 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:48:31.673 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 info mirror/image_22@snap 2024-09-12T06:48:31.721 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:rbd image 'image_22': 2024-09-12T06:48:31.721 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: size 128 MiB in 32 objects 2024-09-12T06:48:31.721 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: order 22 (4 MiB objects) 2024-09-12T06:48:31.721 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: snapshot_count: 1 2024-09-12T06:48:31.721 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: id: 10af5c59572d 2024-09-12T06:48:31.722 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: block_name_prefix: rbd_data.10af5c59572d 2024-09-12T06:48:31.722 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: format: 2 2024-09-12T06:48:31.722 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: features: layering, exclusive-lock, journaling 2024-09-12T06:48:31.722 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: op_features: 2024-09-12T06:48:31.722 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: flags: 2024-09-12T06:48:31.722 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: create_timestamp: Thu Sep 12 06:41:40 2024 2024-09-12T06:48:31.722 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: access_timestamp: Thu Sep 12 06:41:40 2024 2024-09-12T06:48:31.723 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: modify_timestamp: Thu Sep 12 06:42:41 2024 2024-09-12T06:48:31.723 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: protected: False 2024-09-12T06:48:31.723 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: journal: 10af5c59572d 2024-09-12T06:48:31.723 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring state: enabled 2024-09-12T06:48:31.723 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring mode: journal 2024-09-12T06:48:31.723 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring global id: 42f9235e-741e-40f5-9077-c4259b50f9d0 2024-09-12T06:48:31.723 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring primary: false 2024-09-12T06:48:31.725 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:48:31.725 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror image_22 snap 2024-09-12T06:48:31.726 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:31.726 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_22 2024-09-12T06:48:31.726 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:48:31.726 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:48:31.726 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_22.export 2024-09-12T06:48:31.726 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_22.export 2024-09-12T06:48:31.726 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_22.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_22.export 2024-09-12T06:48:31.726 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export image_22@snap /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_22.export 2024-09-12T06:48:31.854 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:48:31.860 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export image_22@snap /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_22.export 2024-09-12T06:48:32.035 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:48:32.041 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_22.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_22.export 2024-09-12T06:48:32.103 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_22.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_22.export 2024-09-12T06:48:32.133 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:48:32.134 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:48:32.134 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_23 2024-09-12T06:48:32.134 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_snap cluster2 mirror image_23 snap 2024-09-12T06:48:32.134 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:48:32.134 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:32.134 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_23 2024-09-12T06:48:32.134 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:48:32.134 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror snap create image_23@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T06:48:32.904 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_started cluster1 mirror image_23 2024-09-12T06:48:32.904 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:48:32.904 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:32.904 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_23 2024-09-12T06:48:32.904 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_state cluster1 mirror image_23 started 2024-09-12T06:48:32.904 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:48:32.904 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:32.904 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_23 2024-09-12T06:48:32.904 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=started 2024-09-12T06:48:32.904 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:48:32.905 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:48:32.905 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:48:33.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_replay_state cluster1 mirror image_23 started 2024-09-12T06:48:33.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:48:33.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:33.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_23 2024-09-12T06:48:33.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=started 2024-09-12T06:48:33.907 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local status_result 2024-09-12T06:48:33.907 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=stopped 2024-09-12T06:48:33.907 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_23 2024-09-12T06:48:33.907 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -i state 2024-09-12T06:48:33.907 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_instance=cluster1 2024-09-12T06:48:33.907 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:48:33.907 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:48:33.907 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance=cluster1 2024-09-12T06:48:33.907 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local loop_instance 2024-09-12T06:48:33.907 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:48:33.907 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sleep 0 2024-09-12T06:48:33.907 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:48:33.907 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ seq 0 1 2024-09-12T06:48:33.910 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:48:33.910 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_23 2024-09-12T06:48:33.910 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:0 2024-09-12T06:48:33.910 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:48:33.910 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:48:33.910 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:48:33.911 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:48:33.911 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:48:33.911 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:48:33.911 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:48:33.911 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:48:33.911 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:48:33.911 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:48:33.911 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:48:33.911 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:48:33.911 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:48:33.911 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:48:33.911 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:0 2024-09-12T06:48:33.911 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:48:33.911 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:48:33.912 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:48:33.912 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:0 2024-09-12T06:48:33.912 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:48:33.912 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:48:33.912 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:48:33.912 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:48:33.912 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T06:48:33.912 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:48:33.912 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:48:33.912 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=0 2024-09-12T06:48:33.912 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=0 2024-09-12T06:48:33.912 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:48:33.928 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:48:33.928 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:48:33.928 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:48:33.928 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_23 2024-09-12T06:48:34.026 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ return 0 2024-09-12T06:48:34.026 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T06:48:34.026 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo ' "state": "Replaying"' 2024-09-12T06:48:34.026 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep -i Replaying 2024-09-12T06:48:34.027 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: "state": "Replaying" 2024-09-12T06:48:34.028 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_state=started 2024-09-12T06:48:34.029 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test started = started 2024-09-12T06:48:34.029 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:48:34.029 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_23 2024-09-12T06:48:34.029 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:48:34.029 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:48:34.029 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:48:34.029 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:48:34.029 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_23 2024-09-12T06:48:34.029 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' journal = journal ']' 2024-09-12T06:48:34.029 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror image_23 2024-09-12T06:48:34.029 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:48:34.029 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:48:34.029 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:48:34.030 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:48:34.030 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_23 2024-09-12T06:48:34.030 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-12T06:48:34.030 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-12T06:48:34.030 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:48:34.030 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:48:34.030 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:48:34.229 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror image_23 2024-09-12T06:48:34.230 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:48:34.230 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:34.230 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_23 2024-09-12T06:48:34.230 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:48:34.230 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n image_23 ']' 2024-09-12T06:48:34.231 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/image_23' 2024-09-12T06:48:34.231 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_23 2024-09-12T06:48:34.231 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:48:34.231 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:48:34.231 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:48:34.231 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:48:34.231 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:48:34.231 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:48:34.231 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:48:34.231 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:48:34.231 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:48:34.235 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:48:34.235 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_23 2024-09-12T06:48:34.235 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:48:34.235 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:48:34.235 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:48:34.235 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:48:34.235 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:48:34.235 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:48:34.236 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:48:34.236 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:48:34.236 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:48:34.236 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:48:34.236 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:48:34.236 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:48:34.236 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:48:34.236 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:48:34.236 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:48:34.236 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:48:34.236 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:48:34.237 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:48:34.237 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:48:34.237 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:48:34.237 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:48:34.237 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:48:34.237 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:48:34.237 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:48:34.237 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:48:34.237 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:48:34.237 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:48:34.237 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:48:34.237 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:48:34.237 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:48:34.251 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:48:34.251 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:48:34.251 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:48:34.251 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_23 2024-09-12T06:48:34.346 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:48:34.346 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror image_23 2024-09-12T06:48:34.346 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:48:34.346 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:48:34.346 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_23 2024-09-12T06:48:34.346 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_23 '' 2024-09-12T06:48:34.346 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:48:34.346 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:48:34.347 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_23 2024-09-12T06:48:34.347 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:48:34.347 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_23.status 2024-09-12T06:48:34.347 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_23.status 2024-09-12T06:48:34.347 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:48:34.347 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_23.status 2024-09-12T06:48:34.347 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_23 2024-09-12T06:48:34.348 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_23.status 2024-09-12T06:48:34.393 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 1 2024-09-12T06:48:34.393 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 1 2024-09-12T06:48:34.393 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:48:34.393 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=3539], [object_number=6, tag_tid=1, entry_tid=3538]]], state=connected] 2024-09-12T06:48:34.393 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=3539], [object_number=6, tag_tid=1, entry_tid=3538]]], state=connected] 2024-09-12T06:48:34.396 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_23.status 2024-09-12T06:48:34.397 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-12T06:48:34.398 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror image_23 2024-09-12T06:48:34.398 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:48:34.398 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:48:34.399 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_23 2024-09-12T06:48:34.399 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_23 '..*' 2024-09-12T06:48:34.399 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:48:34.399 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:48:34.399 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_23 2024-09-12T06:48:34.399 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:48:34.399 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_23.status 2024-09-12T06:48:34.399 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_23.status 2024-09-12T06:48:34.399 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:48:34.400 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_23.status 2024-09-12T06:48:34.400 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_23 2024-09-12T06:48:34.400 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_23.status 2024-09-12T06:48:34.447 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 1 2024-09-12T06:48:34.447 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 1 2024-09-12T06:48:34.447 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:48:34.447 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=3539], [object_number=6, tag_tid=1, entry_tid=3538]]], state=connected] 2024-09-12T06:48:34.447 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=3539], [object_number=6, tag_tid=1, entry_tid=3538]]], state=connected] 2024-09-12T06:48:34.449 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_23.status 2024-09-12T06:48:34.452 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-12T06:48:34.452 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-09-12T06:48:34.452 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:48:34.452 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_snap_present cluster1 mirror image_23 snap 2024-09-12T06:48:34.452 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:48:34.452 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:34.452 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_23 2024-09-12T06:48:34.452 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:48:34.452 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:48:34.452 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T06:48:34.452 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:48:35.452 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 info mirror/image_23@snap 2024-09-12T06:48:35.498 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:rbd image 'image_23': 2024-09-12T06:48:35.498 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: size 128 MiB in 32 objects 2024-09-12T06:48:35.498 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: order 22 (4 MiB objects) 2024-09-12T06:48:35.498 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: snapshot_count: 1 2024-09-12T06:48:35.498 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: id: 10a8f8c297c0 2024-09-12T06:48:35.498 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: block_name_prefix: rbd_data.10a8f8c297c0 2024-09-12T06:48:35.499 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: format: 2 2024-09-12T06:48:35.499 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: features: layering, exclusive-lock, journaling 2024-09-12T06:48:35.499 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: op_features: 2024-09-12T06:48:35.499 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: flags: 2024-09-12T06:48:35.499 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: create_timestamp: Thu Sep 12 06:41:46 2024 2024-09-12T06:48:35.499 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: access_timestamp: Thu Sep 12 06:41:46 2024 2024-09-12T06:48:35.499 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: modify_timestamp: Thu Sep 12 06:42:47 2024 2024-09-12T06:48:35.499 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: protected: False 2024-09-12T06:48:35.499 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: journal: 10a8f8c297c0 2024-09-12T06:48:35.499 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring state: enabled 2024-09-12T06:48:35.499 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring mode: journal 2024-09-12T06:48:35.499 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring global id: fc62c542-aeba-4b48-857a-612ec050e6bd 2024-09-12T06:48:35.499 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring primary: false 2024-09-12T06:48:35.502 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:48:35.502 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror image_23 snap 2024-09-12T06:48:35.502 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:35.502 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_23 2024-09-12T06:48:35.502 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:48:35.502 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:48:35.502 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_23.export 2024-09-12T06:48:35.502 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_23.export 2024-09-12T06:48:35.503 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_23.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_23.export 2024-09-12T06:48:35.503 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export image_23@snap /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_23.export 2024-09-12T06:48:35.654 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:48:35.660 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export image_23@snap /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_23.export 2024-09-12T06:48:35.826 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:48:35.833 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_23.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_23.export 2024-09-12T06:48:35.888 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_23.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_23.export 2024-09-12T06:48:35.920 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:48:35.920 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:48:35.920 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_24 2024-09-12T06:48:35.920 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_snap cluster2 mirror image_24 snap 2024-09-12T06:48:35.920 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:48:35.920 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:35.921 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_24 2024-09-12T06:48:35.921 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:48:35.921 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror snap create image_24@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T06:48:38.915 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_started cluster1 mirror image_24 2024-09-12T06:48:38.916 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:48:38.916 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:38.916 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_24 2024-09-12T06:48:38.916 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_state cluster1 mirror image_24 started 2024-09-12T06:48:38.916 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:48:38.916 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:38.916 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_24 2024-09-12T06:48:38.916 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=started 2024-09-12T06:48:38.916 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:48:38.916 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:48:38.916 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:48:39.820 INFO:tasks.ceph.cluster2.osd.2.smithi152.stderr:2024-09-12T06:48:39.821+0000 7f4789beb640 -1 reset not still connected to 0x56163d519e10 2024-09-12T06:48:39.820 INFO:tasks.ceph.cluster2.osd.2.smithi152.stderr:2024-09-12T06:48:39.821+0000 7f4789beb640 -1 reset not still connected to 0x56163df360d0 2024-09-12T06:48:39.820 INFO:tasks.ceph.cluster2.osd.2.smithi152.stderr:2024-09-12T06:48:39.821+0000 7f4789beb640 -1 reset not still connected to 0x56163df36270 2024-09-12T06:48:39.918 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_replay_state cluster1 mirror image_24 started 2024-09-12T06:48:39.918 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:48:39.918 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:39.918 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_24 2024-09-12T06:48:39.918 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=started 2024-09-12T06:48:39.918 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local status_result 2024-09-12T06:48:39.919 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=stopped 2024-09-12T06:48:39.919 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_24 2024-09-12T06:48:39.919 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -i state 2024-09-12T06:48:39.919 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_instance=cluster1 2024-09-12T06:48:39.919 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:48:39.919 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:48:39.919 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance=cluster1 2024-09-12T06:48:39.919 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local loop_instance 2024-09-12T06:48:39.919 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:48:39.919 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sleep 0 2024-09-12T06:48:39.919 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:48:39.919 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ seq 0 1 2024-09-12T06:48:39.923 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:48:39.923 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_24 2024-09-12T06:48:39.923 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:0 2024-09-12T06:48:39.923 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:48:39.923 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:48:39.923 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:48:39.923 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:48:39.923 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:48:39.923 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:48:39.923 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:48:39.923 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:48:39.924 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:48:39.924 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:48:39.924 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:48:39.924 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:48:39.924 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:48:39.924 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:48:39.924 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:0 2024-09-12T06:48:39.924 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:48:39.924 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:48:39.924 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:48:39.924 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:0 2024-09-12T06:48:39.924 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:48:39.925 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:48:39.925 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:48:39.925 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:48:39.925 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T06:48:39.925 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:48:39.925 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:48:39.925 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=0 2024-09-12T06:48:39.925 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=0 2024-09-12T06:48:39.925 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:48:39.941 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:48:39.941 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:48:39.941 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:48:39.941 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_24 2024-09-12T06:48:40.034 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2024-09-12T06:48:40.035 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T06:48:40.035 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:admin_socket: invalid command 2024-09-12T06:48:40.038 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:48:40.039 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_24 2024-09-12T06:48:40.039 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:1 2024-09-12T06:48:40.039 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:48:40.039 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:48:40.039 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T06:48:40.039 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:1 2024-09-12T06:48:40.039 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:48:40.039 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:48:40.040 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:48:40.040 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:48:40.040 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T06:48:40.040 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:48:40.040 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:48:40.040 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=1 2024-09-12T06:48:40.040 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:48:40.040 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T06:48:40.040 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:1 2024-09-12T06:48:40.040 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:48:40.040 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:48:40.040 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T06:48:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:1 2024-09-12T06:48:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:48:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:48:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:48:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:48:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T06:48:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:48:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:48:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=1 2024-09-12T06:48:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=1 2024-09-12T06:48:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T06:48:40.057 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:48:40.058 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:48:40.058 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:48:40.058 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_24 2024-09-12T06:48:40.158 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ return 0 2024-09-12T06:48:40.159 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T06:48:40.159 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo ' "state": "Replaying"' 2024-09-12T06:48:40.159 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep -i Replaying 2024-09-12T06:48:40.161 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: "state": "Replaying" 2024-09-12T06:48:40.162 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_state=started 2024-09-12T06:48:40.162 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test started = started 2024-09-12T06:48:40.162 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:48:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_24 2024-09-12T06:48:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:48:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:48:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:48:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:48:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_24 2024-09-12T06:48:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' journal = journal ']' 2024-09-12T06:48:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror image_24 2024-09-12T06:48:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:48:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:48:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:48:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:48:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_24 2024-09-12T06:48:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-12T06:48:40.164 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-12T06:48:40.164 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:48:40.164 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:48:40.164 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:48:40.362 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror image_24 2024-09-12T06:48:40.363 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:48:40.363 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:40.363 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_24 2024-09-12T06:48:40.363 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:48:40.363 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n image_24 ']' 2024-09-12T06:48:40.363 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/image_24' 2024-09-12T06:48:40.363 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_24 2024-09-12T06:48:40.363 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:48:40.363 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:48:40.363 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:48:40.363 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:48:40.363 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:48:40.363 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:48:40.364 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:48:40.364 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:48:40.364 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:48:40.367 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:48:40.367 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_24 2024-09-12T06:48:40.367 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:48:40.367 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:48:40.367 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:48:40.367 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:48:40.367 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:48:40.367 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:48:40.367 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:48:40.367 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:48:40.367 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:48:40.368 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:48:40.368 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:48:40.368 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:48:40.368 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:48:40.368 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:48:40.368 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:48:40.368 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:48:40.368 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:48:40.368 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:48:40.368 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:48:40.368 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:48:40.368 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:48:40.368 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:48:40.368 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:48:40.369 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:48:40.369 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:48:40.369 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:48:40.369 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:48:40.369 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:48:40.369 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:48:40.369 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:48:40.384 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:48:40.384 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:48:40.385 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:48:40.385 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_24 2024-09-12T06:48:40.469 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2024-09-12T06:48:40.470 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd mirror flush : flush rbd mirror 2024-09-12T06:48:40.470 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:admin_socket: invalid command 2024-09-12T06:48:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:48:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_24 2024-09-12T06:48:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:1 2024-09-12T06:48:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:48:40.474 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:48:40.474 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-12T06:48:40.474 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:1 2024-09-12T06:48:40.474 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:48:40.474 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:48:40.474 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:48:40.474 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=1 2024-09-12T06:48:40.474 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-12T06:48:40.474 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:48:40.474 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:48:40.474 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=1 2024-09-12T06:48:40.474 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:48:40.474 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-12T06:48:40.474 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:1 2024-09-12T06:48:40.475 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:48:40.475 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:48:40.475 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T06:48:40.475 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:1 2024-09-12T06:48:40.475 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:48:40.475 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:48:40.475 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:48:40.475 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:48:40.475 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T06:48:40.475 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:48:40.475 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:48:40.475 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=1 2024-09-12T06:48:40.475 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:48:40.475 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T06:48:40.492 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:48:40.492 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:48:40.492 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:48:40.492 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_24 2024-09-12T06:48:40.588 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:48:40.588 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror image_24 2024-09-12T06:48:40.588 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:48:40.588 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:48:40.588 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_24 2024-09-12T06:48:40.589 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_24 '' 2024-09-12T06:48:40.589 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:48:40.589 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:48:40.589 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_24 2024-09-12T06:48:40.589 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:48:40.589 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_24.status 2024-09-12T06:48:40.589 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_24.status 2024-09-12T06:48:40.589 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:48:40.590 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_24.status 2024-09-12T06:48:40.590 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_24 2024-09-12T06:48:40.590 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_24.status 2024-09-12T06:48:40.639 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 2 2024-09-12T06:48:40.639 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 2 2024-09-12T06:48:40.639 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:48:40.639 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9443], [object_number=10, tag_tid=1, entry_tid=9442]]], state=connected] 2024-09-12T06:48:40.639 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9443], [object_number=10, tag_tid=1, entry_tid=9442]]], state=connected] 2024-09-12T06:48:40.642 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_24.status 2024-09-12T06:48:40.644 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-09-12T06:48:40.644 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror image_24 2024-09-12T06:48:40.644 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:48:40.644 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:48:40.644 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_24 2024-09-12T06:48:40.645 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_24 '..*' 2024-09-12T06:48:40.645 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:48:40.645 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:48:40.645 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_24 2024-09-12T06:48:40.645 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:48:40.645 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_24.status 2024-09-12T06:48:40.645 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_24.status 2024-09-12T06:48:40.645 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:48:40.646 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_24.status 2024-09-12T06:48:40.646 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_24 2024-09-12T06:48:40.646 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_24.status 2024-09-12T06:48:40.690 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 2 2024-09-12T06:48:40.690 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 2 2024-09-12T06:48:40.691 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:48:40.691 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9443], [object_number=10, tag_tid=1, entry_tid=9442]]], state=connected] 2024-09-12T06:48:40.691 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9443], [object_number=10, tag_tid=1, entry_tid=9442]]], state=connected] 2024-09-12T06:48:40.693 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_24.status 2024-09-12T06:48:40.696 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-09-12T06:48:40.696 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-09-12T06:48:40.696 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:48:40.696 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_snap_present cluster1 mirror image_24 snap 2024-09-12T06:48:40.696 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:48:40.696 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:40.696 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_24 2024-09-12T06:48:40.696 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:48:40.696 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:48:40.697 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T06:48:40.697 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:48:41.696 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 info mirror/image_24@snap 2024-09-12T06:48:41.743 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:rbd image 'image_24': 2024-09-12T06:48:41.743 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: size 128 MiB in 32 objects 2024-09-12T06:48:41.743 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: order 22 (4 MiB objects) 2024-09-12T06:48:41.743 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: snapshot_count: 1 2024-09-12T06:48:41.744 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: id: 10af69a20361 2024-09-12T06:48:41.744 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: block_name_prefix: rbd_data.10af69a20361 2024-09-12T06:48:41.744 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: format: 2 2024-09-12T06:48:41.744 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: features: layering, exclusive-lock, journaling 2024-09-12T06:48:41.744 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: op_features: 2024-09-12T06:48:41.744 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: flags: 2024-09-12T06:48:41.744 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: create_timestamp: Thu Sep 12 06:41:52 2024 2024-09-12T06:48:41.744 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: access_timestamp: Thu Sep 12 06:41:52 2024 2024-09-12T06:48:41.744 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: modify_timestamp: Thu Sep 12 06:42:53 2024 2024-09-12T06:48:41.744 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: protected: False 2024-09-12T06:48:41.744 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: journal: 10af69a20361 2024-09-12T06:48:41.744 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring state: enabled 2024-09-12T06:48:41.744 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring mode: journal 2024-09-12T06:48:41.745 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring global id: bd9d254f-9121-4598-822f-ebeec662219d 2024-09-12T06:48:41.745 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring primary: false 2024-09-12T06:48:41.747 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:48:41.747 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror image_24 snap 2024-09-12T06:48:41.747 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:41.747 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_24 2024-09-12T06:48:41.748 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:48:41.748 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:48:41.748 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_24.export 2024-09-12T06:48:41.748 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_24.export 2024-09-12T06:48:41.748 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_24.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_24.export 2024-09-12T06:48:41.748 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export image_24@snap /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_24.export 2024-09-12T06:48:41.901 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:48:41.907 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export image_24@snap /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_24.export 2024-09-12T06:48:42.105 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:48:42.112 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_24.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_24.export 2024-09-12T06:48:42.167 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_24.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_24.export 2024-09-12T06:48:42.199 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:48:42.199 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:48:42.199 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_25 2024-09-12T06:48:42.199 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_snap cluster2 mirror image_25 snap 2024-09-12T06:48:42.199 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:48:42.199 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:42.199 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_25 2024-09-12T06:48:42.199 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:48:42.199 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror snap create image_25@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T06:48:44.920 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_started cluster1 mirror image_25 2024-09-12T06:48:44.920 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:48:44.920 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:44.920 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_25 2024-09-12T06:48:44.921 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_state cluster1 mirror image_25 started 2024-09-12T06:48:44.921 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:48:44.921 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:44.921 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_25 2024-09-12T06:48:44.921 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=started 2024-09-12T06:48:44.921 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:48:44.921 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:48:44.921 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:48:45.921 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_replay_state cluster1 mirror image_25 started 2024-09-12T06:48:45.922 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:48:45.922 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:45.922 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_25 2024-09-12T06:48:45.922 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=started 2024-09-12T06:48:45.922 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local status_result 2024-09-12T06:48:45.922 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=stopped 2024-09-12T06:48:45.923 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_25 2024-09-12T06:48:45.923 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -i state 2024-09-12T06:48:45.923 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_instance=cluster1 2024-09-12T06:48:45.923 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:48:45.923 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:48:45.923 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance=cluster1 2024-09-12T06:48:45.924 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local loop_instance 2024-09-12T06:48:45.924 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:48:45.924 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sleep 0 2024-09-12T06:48:45.924 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:48:45.924 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ seq 0 1 2024-09-12T06:48:45.927 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:48:45.927 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_25 2024-09-12T06:48:45.927 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:0 2024-09-12T06:48:45.927 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:48:45.927 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:48:45.927 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:48:45.927 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:48:45.927 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:48:45.927 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:48:45.927 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:48:45.928 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:48:45.928 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:48:45.928 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:48:45.928 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:48:45.928 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:48:45.928 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:48:45.928 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:48:45.928 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:0 2024-09-12T06:48:45.928 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:48:45.928 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:48:45.928 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:48:45.928 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:0 2024-09-12T06:48:45.928 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:48:45.929 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:48:45.929 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:48:45.929 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:48:45.929 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T06:48:45.929 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:48:45.929 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:48:45.929 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=0 2024-09-12T06:48:45.929 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=0 2024-09-12T06:48:45.929 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:48:45.946 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:48:45.946 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:48:45.947 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:48:45.947 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_25 2024-09-12T06:48:46.048 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ return 0 2024-09-12T06:48:46.048 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T06:48:46.048 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo ' "state": "Replaying"' 2024-09-12T06:48:46.049 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep -i Replaying 2024-09-12T06:48:46.049 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: "state": "Replaying" 2024-09-12T06:48:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_state=started 2024-09-12T06:48:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test started = started 2024-09-12T06:48:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:48:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_25 2024-09-12T06:48:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:48:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:48:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:48:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:48:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_25 2024-09-12T06:48:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' journal = journal ']' 2024-09-12T06:48:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror image_25 2024-09-12T06:48:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:48:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:48:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:48:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:48:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_25 2024-09-12T06:48:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-12T06:48:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-12T06:48:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:48:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:48:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:48:46.252 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror image_25 2024-09-12T06:48:46.252 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:48:46.252 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:46.252 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_25 2024-09-12T06:48:46.252 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:48:46.252 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n image_25 ']' 2024-09-12T06:48:46.252 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/image_25' 2024-09-12T06:48:46.252 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_25 2024-09-12T06:48:46.253 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:48:46.253 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:48:46.253 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:48:46.253 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:48:46.253 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:48:46.253 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:48:46.253 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:48:46.253 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:48:46.256 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:48:46.256 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:48:46.256 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_25 2024-09-12T06:48:46.256 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:48:46.256 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:48:46.256 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:48:46.256 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:48:46.257 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:48:46.257 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:48:46.257 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:48:46.257 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:48:46.257 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:48:46.257 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:48:46.257 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:48:46.257 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:48:46.257 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:48:46.257 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:48:46.257 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:48:46.257 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:48:46.257 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:48:46.258 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:48:46.258 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:48:46.258 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:48:46.258 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:48:46.258 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:48:46.258 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:48:46.258 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:48:46.258 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:48:46.258 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:48:46.258 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:48:46.258 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:48:46.258 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:48:46.258 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:48:46.274 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:48:46.274 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:48:46.274 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:48:46.274 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_25 2024-09-12T06:48:46.376 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:48:46.376 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror image_25 2024-09-12T06:48:46.376 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:48:46.376 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:48:46.376 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_25 2024-09-12T06:48:46.376 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_25 '' 2024-09-12T06:48:46.376 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:48:46.376 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:48:46.376 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_25 2024-09-12T06:48:46.377 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:48:46.377 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_25.status 2024-09-12T06:48:46.377 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_25.status 2024-09-12T06:48:46.377 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:48:46.377 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_25.status 2024-09-12T06:48:46.377 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_25 2024-09-12T06:48:46.377 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_25.status 2024-09-12T06:48:46.422 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 3 2024-09-12T06:48:46.423 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 3 2024-09-12T06:48:46.423 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:48:46.423 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=10959], [object_number=14, tag_tid=1, entry_tid=10958]]], state=connected] 2024-09-12T06:48:46.423 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=10959], [object_number=14, tag_tid=1, entry_tid=10958]]], state=connected] 2024-09-12T06:48:46.425 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_25.status 2024-09-12T06:48:46.428 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-09-12T06:48:46.428 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror image_25 2024-09-12T06:48:46.428 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:48:46.428 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:48:46.428 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_25 2024-09-12T06:48:46.428 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_25 '..*' 2024-09-12T06:48:46.428 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:48:46.428 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:48:46.428 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_25 2024-09-12T06:48:46.428 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:48:46.428 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_25.status 2024-09-12T06:48:46.429 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_25.status 2024-09-12T06:48:46.429 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:48:46.429 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_25.status 2024-09-12T06:48:46.429 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_25 2024-09-12T06:48:46.429 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_25.status 2024-09-12T06:48:46.474 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 3 2024-09-12T06:48:46.475 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 3 2024-09-12T06:48:46.475 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:48:46.475 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=10959], [object_number=14, tag_tid=1, entry_tid=10958]]], state=connected] 2024-09-12T06:48:46.475 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=10959], [object_number=14, tag_tid=1, entry_tid=10958]]], state=connected] 2024-09-12T06:48:46.477 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_25.status 2024-09-12T06:48:46.479 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-09-12T06:48:46.479 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=1' 2024-09-12T06:48:46.479 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:48:46.480 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_snap_present cluster1 mirror image_25 snap 2024-09-12T06:48:46.480 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:48:46.480 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:46.480 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_25 2024-09-12T06:48:46.480 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:48:46.480 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:48:46.480 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T06:48:46.480 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:48:47.480 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 info mirror/image_25@snap 2024-09-12T06:48:47.532 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:rbd image 'image_25': 2024-09-12T06:48:47.532 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: size 128 MiB in 32 objects 2024-09-12T06:48:47.532 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: order 22 (4 MiB objects) 2024-09-12T06:48:47.532 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: snapshot_count: 1 2024-09-12T06:48:47.532 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: id: 10a85de718c8 2024-09-12T06:48:47.532 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: block_name_prefix: rbd_data.10a85de718c8 2024-09-12T06:48:47.533 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: format: 2 2024-09-12T06:48:47.533 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: features: layering, exclusive-lock, journaling 2024-09-12T06:48:47.533 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: op_features: 2024-09-12T06:48:47.533 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: flags: 2024-09-12T06:48:47.533 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: create_timestamp: Thu Sep 12 06:42:01 2024 2024-09-12T06:48:47.533 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: access_timestamp: Thu Sep 12 06:42:01 2024 2024-09-12T06:48:47.533 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: modify_timestamp: Thu Sep 12 06:43:02 2024 2024-09-12T06:48:47.533 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: protected: False 2024-09-12T06:48:47.533 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: journal: 10a85de718c8 2024-09-12T06:48:47.533 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring state: enabled 2024-09-12T06:48:47.534 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring mode: journal 2024-09-12T06:48:47.534 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring global id: 159e3cc2-9797-439e-a2a5-7a42ac01044c 2024-09-12T06:48:47.534 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring primary: false 2024-09-12T06:48:47.536 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:48:47.536 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror image_25 snap 2024-09-12T06:48:47.536 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:47.536 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_25 2024-09-12T06:48:47.536 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:48:47.536 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:48:47.537 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_25.export 2024-09-12T06:48:47.537 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_25.export 2024-09-12T06:48:47.537 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_25.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_25.export 2024-09-12T06:48:47.537 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export image_25@snap /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_25.export 2024-09-12T06:48:47.691 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:48:47.697 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export image_25@snap /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_25.export 2024-09-12T06:48:47.934 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:48:47.940 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_25.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_25.export 2024-09-12T06:48:47.996 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_25.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_25.export 2024-09-12T06:48:48.029 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:48:48.030 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:48:48.030 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_26 2024-09-12T06:48:48.030 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_snap cluster2 mirror image_26 snap 2024-09-12T06:48:48.030 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:48:48.030 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:48.030 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_26 2024-09-12T06:48:48.030 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:48:48.030 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror snap create image_26@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T06:48:50.928 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_started cluster1 mirror image_26 2024-09-12T06:48:50.928 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:48:50.928 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:50.928 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_26 2024-09-12T06:48:50.928 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_state cluster1 mirror image_26 started 2024-09-12T06:48:50.928 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:48:50.929 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:50.929 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_26 2024-09-12T06:48:50.929 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=started 2024-09-12T06:48:50.929 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:48:50.929 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:48:50.929 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:48:51.937 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_replay_state cluster1 mirror image_26 started 2024-09-12T06:48:51.938 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:48:51.938 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:51.938 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_26 2024-09-12T06:48:51.938 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=started 2024-09-12T06:48:51.938 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local status_result 2024-09-12T06:48:51.938 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=stopped 2024-09-12T06:48:51.939 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_26 2024-09-12T06:48:51.939 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_instance=cluster1 2024-09-12T06:48:51.939 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -i state 2024-09-12T06:48:51.939 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:48:51.939 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:48:51.939 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance=cluster1 2024-09-12T06:48:51.940 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local loop_instance 2024-09-12T06:48:51.940 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:48:51.940 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sleep 0 2024-09-12T06:48:51.940 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:48:51.940 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ seq 0 1 2024-09-12T06:48:51.940 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:48:51.941 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_26 2024-09-12T06:48:51.941 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:0 2024-09-12T06:48:51.941 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:48:51.941 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:48:51.941 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:48:51.941 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:48:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:48:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:48:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:48:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:48:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:48:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:48:51.943 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:48:51.943 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:48:51.943 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:48:51.943 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:48:51.943 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:0 2024-09-12T06:48:51.943 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:48:51.944 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:48:51.944 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:48:51.944 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:0 2024-09-12T06:48:51.944 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:48:51.944 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:48:51.944 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:48:51.945 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:48:51.945 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T06:48:51.945 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:48:51.945 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:48:51.945 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=0 2024-09-12T06:48:51.945 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=0 2024-09-12T06:48:51.945 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:48:51.953 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:48:51.953 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:48:51.953 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:48:51.954 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_26 2024-09-12T06:48:52.045 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2024-09-12T06:48:52.046 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T06:48:52.046 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:admin_socket: invalid command 2024-09-12T06:48:52.049 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:48:52.049 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_26 2024-09-12T06:48:52.049 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:1 2024-09-12T06:48:52.049 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:48:52.049 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:48:52.049 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T06:48:52.050 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:1 2024-09-12T06:48:52.050 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:48:52.050 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:48:52.050 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:48:52.050 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:48:52.050 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T06:48:52.050 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:48:52.050 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:48:52.050 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=1 2024-09-12T06:48:52.050 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:48:52.050 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T06:48:52.050 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:1 2024-09-12T06:48:52.050 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:48:52.050 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:48:52.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T06:48:52.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:1 2024-09-12T06:48:52.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:48:52.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:48:52.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:48:52.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:48:52.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T06:48:52.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:48:52.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:48:52.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=1 2024-09-12T06:48:52.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=1 2024-09-12T06:48:52.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T06:48:52.069 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:48:52.069 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:48:52.069 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:48:52.069 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_26 2024-09-12T06:48:52.170 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ return 0 2024-09-12T06:48:52.171 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T06:48:52.171 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo ' "state": "Replaying"' 2024-09-12T06:48:52.171 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep -i Replaying 2024-09-12T06:48:52.172 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: "state": "Replaying" 2024-09-12T06:48:52.173 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_state=started 2024-09-12T06:48:52.173 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test started = started 2024-09-12T06:48:52.173 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:48:52.173 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_26 2024-09-12T06:48:52.173 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:48:52.174 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:48:52.174 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:48:52.174 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:48:52.174 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_26 2024-09-12T06:48:52.174 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' journal = journal ']' 2024-09-12T06:48:52.174 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror image_26 2024-09-12T06:48:52.174 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:48:52.174 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:48:52.174 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:48:52.174 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:48:52.174 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_26 2024-09-12T06:48:52.174 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-12T06:48:52.174 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-12T06:48:52.174 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:48:52.175 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:48:52.175 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:48:52.374 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror image_26 2024-09-12T06:48:52.374 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:48:52.374 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:52.375 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_26 2024-09-12T06:48:52.375 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:48:52.375 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n image_26 ']' 2024-09-12T06:48:52.375 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/image_26' 2024-09-12T06:48:52.375 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_26 2024-09-12T06:48:52.375 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:48:52.375 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:48:52.375 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:48:52.375 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:48:52.375 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:48:52.375 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:48:52.375 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:48:52.375 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:48:52.376 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:48:52.379 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:48:52.379 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_26 2024-09-12T06:48:52.379 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:48:52.379 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:48:52.379 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:48:52.379 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:48:52.379 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:48:52.379 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:48:52.379 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:48:52.380 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:48:52.380 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:48:52.380 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:48:52.380 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:48:52.380 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:48:52.380 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:48:52.380 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:48:52.380 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:48:52.380 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:48:52.380 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:48:52.380 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:48:52.380 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:48:52.380 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:48:52.381 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:48:52.381 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:48:52.381 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:48:52.381 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:48:52.381 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:48:52.381 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:48:52.381 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:48:52.381 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:48:52.381 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:48:52.381 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:48:52.396 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:48:52.396 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:48:52.396 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:48:52.396 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_26 2024-09-12T06:48:52.482 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2024-09-12T06:48:52.483 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd mirror flush : flush rbd mirror 2024-09-12T06:48:52.483 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:admin_socket: invalid command 2024-09-12T06:48:52.486 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:48:52.486 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_26 2024-09-12T06:48:52.486 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:1 2024-09-12T06:48:52.486 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:48:52.487 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:48:52.487 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-12T06:48:52.487 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:1 2024-09-12T06:48:52.487 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:48:52.487 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:48:52.487 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:48:52.487 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=1 2024-09-12T06:48:52.487 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-12T06:48:52.487 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:48:52.487 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:48:52.487 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=1 2024-09-12T06:48:52.487 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:48:52.488 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-12T06:48:52.488 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:1 2024-09-12T06:48:52.488 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:48:52.488 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:48:52.488 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T06:48:52.488 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:1 2024-09-12T06:48:52.488 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:48:52.488 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:48:52.488 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:48:52.488 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:48:52.488 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T06:48:52.488 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:48:52.488 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:48:52.488 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=1 2024-09-12T06:48:52.489 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:48:52.489 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T06:48:52.504 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:48:52.505 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:48:52.505 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:48:52.505 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_26 2024-09-12T06:48:52.601 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:48:52.601 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror image_26 2024-09-12T06:48:52.601 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:48:52.601 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:48:52.601 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_26 2024-09-12T06:48:52.602 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_26 '' 2024-09-12T06:48:52.602 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:48:52.602 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:48:52.602 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_26 2024-09-12T06:48:52.602 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:48:52.602 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_26.status 2024-09-12T06:48:52.602 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_26.status 2024-09-12T06:48:52.602 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:48:52.603 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_26.status 2024-09-12T06:48:52.603 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_26 2024-09-12T06:48:52.603 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_26.status 2024-09-12T06:48:52.648 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 2 2024-09-12T06:48:52.648 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 2 2024-09-12T06:48:52.648 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:48:52.648 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9827], [object_number=10, tag_tid=1, entry_tid=9826]]], state=connected] 2024-09-12T06:48:52.648 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9827], [object_number=10, tag_tid=1, entry_tid=9826]]], state=connected] 2024-09-12T06:48:52.651 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_26.status 2024-09-12T06:48:52.654 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-09-12T06:48:52.654 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror image_26 2024-09-12T06:48:52.654 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:48:52.654 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:48:52.654 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_26 2024-09-12T06:48:52.654 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_26 '..*' 2024-09-12T06:48:52.654 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:48:52.654 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:48:52.654 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_26 2024-09-12T06:48:52.654 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:48:52.654 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_26.status 2024-09-12T06:48:52.654 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_26.status 2024-09-12T06:48:52.655 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:48:52.655 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_26.status 2024-09-12T06:48:52.655 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_26 2024-09-12T06:48:52.655 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_26.status 2024-09-12T06:48:52.700 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 2 2024-09-12T06:48:52.701 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 2 2024-09-12T06:48:52.701 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:48:52.701 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9827], [object_number=10, tag_tid=1, entry_tid=9826]]], state=connected] 2024-09-12T06:48:52.701 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9827], [object_number=10, tag_tid=1, entry_tid=9826]]], state=connected] 2024-09-12T06:48:52.766 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_26.status 2024-09-12T06:48:52.769 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-09-12T06:48:52.769 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-09-12T06:48:52.769 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:48:52.769 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_snap_present cluster1 mirror image_26 snap 2024-09-12T06:48:52.769 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:48:52.769 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:52.769 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_26 2024-09-12T06:48:52.769 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:48:52.769 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:48:52.769 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T06:48:52.769 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:48:53.769 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 info mirror/image_26@snap 2024-09-12T06:48:53.821 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:rbd image 'image_26': 2024-09-12T06:48:53.821 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: size 128 MiB in 32 objects 2024-09-12T06:48:53.821 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: order 22 (4 MiB objects) 2024-09-12T06:48:53.821 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: snapshot_count: 1 2024-09-12T06:48:53.821 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: id: 10af507cda1f 2024-09-12T06:48:53.821 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: block_name_prefix: rbd_data.10af507cda1f 2024-09-12T06:48:53.821 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: format: 2 2024-09-12T06:48:53.821 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: features: layering, exclusive-lock, journaling 2024-09-12T06:48:53.821 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: op_features: 2024-09-12T06:48:53.821 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: flags: 2024-09-12T06:48:53.821 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: create_timestamp: Thu Sep 12 06:42:13 2024 2024-09-12T06:48:53.821 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: access_timestamp: Thu Sep 12 06:42:13 2024 2024-09-12T06:48:53.822 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: modify_timestamp: Thu Sep 12 06:43:14 2024 2024-09-12T06:48:53.822 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: protected: False 2024-09-12T06:48:53.822 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: journal: 10af507cda1f 2024-09-12T06:48:53.822 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring state: enabled 2024-09-12T06:48:53.822 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring mode: journal 2024-09-12T06:48:53.822 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring global id: 019ca26b-0efd-44bf-ad46-9bc886799076 2024-09-12T06:48:53.822 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring primary: false 2024-09-12T06:48:53.825 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:48:53.825 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror image_26 snap 2024-09-12T06:48:53.825 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:53.825 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_26 2024-09-12T06:48:53.826 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:48:53.826 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:48:53.826 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_26.export 2024-09-12T06:48:53.826 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_26.export 2024-09-12T06:48:53.826 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_26.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_26.export 2024-09-12T06:48:53.826 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export image_26@snap /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_26.export 2024-09-12T06:48:53.979 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:48:53.986 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export image_26@snap /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_26.export 2024-09-12T06:48:54.221 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:48:54.227 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_26.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_26.export 2024-09-12T06:48:54.283 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_26.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_26.export 2024-09-12T06:48:54.315 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:48:54.315 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:48:54.315 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_27 2024-09-12T06:48:54.315 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_snap cluster2 mirror image_27 snap 2024-09-12T06:48:54.315 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:48:54.315 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:54.315 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_27 2024-09-12T06:48:54.315 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:48:54.315 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror snap create image_27@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T06:48:54.822 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:48:54.823+0000 7fac12a52640 -1 reset not still connected to 0x55749b2be410 2024-09-12T06:48:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_started cluster1 mirror image_27 2024-09-12T06:48:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:48:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_27 2024-09-12T06:48:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_state cluster1 mirror image_27 started 2024-09-12T06:48:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:48:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_27 2024-09-12T06:48:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=started 2024-09-12T06:48:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:48:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:48:55.947 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:48:56.948 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_replay_state cluster1 mirror image_27 started 2024-09-12T06:48:56.948 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:48:56.948 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:56.948 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_27 2024-09-12T06:48:56.948 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=started 2024-09-12T06:48:56.948 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local status_result 2024-09-12T06:48:56.948 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=stopped 2024-09-12T06:48:56.948 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_27 2024-09-12T06:48:56.948 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_instance=cluster1 2024-09-12T06:48:56.949 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:48:56.949 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -i state 2024-09-12T06:48:56.949 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:48:56.949 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance=cluster1 2024-09-12T06:48:56.949 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local loop_instance 2024-09-12T06:48:56.949 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:48:56.949 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sleep 0 2024-09-12T06:48:56.949 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:48:56.949 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ seq 0 1 2024-09-12T06:48:56.952 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:48:56.952 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_27 2024-09-12T06:48:56.952 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:0 2024-09-12T06:48:56.952 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:48:56.952 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:48:56.953 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:48:56.953 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:48:56.953 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:48:56.953 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:48:56.953 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:48:56.953 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:48:56.953 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:48:56.953 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:48:56.953 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:48:56.953 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:48:56.953 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:48:56.953 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:48:56.953 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:0 2024-09-12T06:48:56.953 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:48:56.954 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:48:56.954 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:48:56.954 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:0 2024-09-12T06:48:56.954 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:48:56.954 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:48:56.954 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:48:56.954 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:48:56.954 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T06:48:56.954 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:48:56.954 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:48:56.954 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=0 2024-09-12T06:48:56.954 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=0 2024-09-12T06:48:56.954 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:48:56.969 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:48:56.970 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:48:56.970 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:48:56.970 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_27 2024-09-12T06:48:57.069 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ return 0 2024-09-12T06:48:57.069 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T06:48:57.069 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo ' "state": "Replaying"' 2024-09-12T06:48:57.069 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep -i Replaying 2024-09-12T06:48:57.072 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: "state": "Replaying" 2024-09-12T06:48:57.074 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_state=started 2024-09-12T06:48:57.074 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test started = started 2024-09-12T06:48:57.074 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:48:57.074 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_27 2024-09-12T06:48:57.075 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:48:57.075 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:48:57.075 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:48:57.075 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:48:57.075 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_27 2024-09-12T06:48:57.075 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' journal = journal ']' 2024-09-12T06:48:57.075 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror image_27 2024-09-12T06:48:57.075 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:48:57.075 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:48:57.075 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:48:57.075 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:48:57.075 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_27 2024-09-12T06:48:57.076 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-12T06:48:57.076 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-12T06:48:57.076 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:48:57.076 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:48:57.076 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:48:57.273 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror image_27 2024-09-12T06:48:57.277 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:48:57.277 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:57.277 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_27 2024-09-12T06:48:57.277 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:48:57.277 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n image_27 ']' 2024-09-12T06:48:57.277 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/image_27' 2024-09-12T06:48:57.277 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_27 2024-09-12T06:48:57.277 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:48:57.277 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:48:57.277 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:48:57.277 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:48:57.277 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:48:57.278 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:48:57.278 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:48:57.278 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:48:57.278 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:48:57.281 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:48:57.281 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_27 2024-09-12T06:48:57.281 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:48:57.281 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:48:57.281 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:48:57.281 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:48:57.281 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:48:57.281 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:48:57.281 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:48:57.281 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:48:57.281 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:48:57.282 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:48:57.282 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:48:57.282 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:48:57.282 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:48:57.282 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:48:57.282 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:48:57.282 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:48:57.282 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:48:57.282 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:48:57.282 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:48:57.282 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:48:57.282 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:48:57.282 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:48:57.282 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:48:57.283 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:48:57.283 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:48:57.283 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:48:57.283 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:48:57.283 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:48:57.283 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:48:57.283 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:48:57.296 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:48:57.296 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:48:57.296 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:48:57.296 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_27 2024-09-12T06:48:57.389 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:48:57.389 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror image_27 2024-09-12T06:48:57.389 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:48:57.389 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:48:57.389 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_27 2024-09-12T06:48:57.389 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_27 '' 2024-09-12T06:48:57.390 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:48:57.390 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:48:57.390 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_27 2024-09-12T06:48:57.390 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:48:57.390 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_27.status 2024-09-12T06:48:57.390 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_27.status 2024-09-12T06:48:57.390 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:48:57.391 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_27.status 2024-09-12T06:48:57.391 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_27 2024-09-12T06:48:57.391 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_27.status 2024-09-12T06:48:57.437 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 1 2024-09-12T06:48:57.437 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 2 2024-09-12T06:48:57.438 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:48:57.438 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6783], [object_number=6, tag_tid=1, entry_tid=6782]]], state=connected] 2024-09-12T06:48:57.438 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=7, tag_tid=1, entry_tid=6783], [object_number=6, tag_tid=1, entry_tid=6782], [object_number=5, tag_tid=1, entry_tid=6781], [object_number=4, tag_tid=1, entry_tid=6780]]], state=connected] 2024-09-12T06:48:57.440 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_27.status 2024-09-12T06:48:57.443 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-09-12T06:48:57.443 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror image_27 2024-09-12T06:48:57.443 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:48:57.443 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:48:57.444 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_27 2024-09-12T06:48:57.444 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_27 '..*' 2024-09-12T06:48:57.444 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:48:57.444 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:48:57.444 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_27 2024-09-12T06:48:57.444 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:48:57.444 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_27.status 2024-09-12T06:48:57.444 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_27.status 2024-09-12T06:48:57.444 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:48:57.445 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_27.status 2024-09-12T06:48:57.445 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_27 2024-09-12T06:48:57.445 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_27.status 2024-09-12T06:48:57.491 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 1 2024-09-12T06:48:57.491 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 2 2024-09-12T06:48:57.491 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:48:57.491 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6783], [object_number=6, tag_tid=1, entry_tid=6782]]], state=connected] 2024-09-12T06:48:57.491 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=7, tag_tid=1, entry_tid=6783], [object_number=6, tag_tid=1, entry_tid=6782], [object_number=5, tag_tid=1, entry_tid=6781], [object_number=4, tag_tid=1, entry_tid=6780]]], state=connected] 2024-09-12T06:48:57.494 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_27.status 2024-09-12T06:48:57.496 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=7, tag_tid=1, entry_tid=6783' 2024-09-12T06:48:57.496 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=7, tag_tid=1, entry_tid=6783' 2024-09-12T06:48:57.496 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 'object_number=7, tag_tid=1, entry_tid=6783' '!=' '' 2024-09-12T06:48:57.496 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ break 2024-09-12T06:48:57.496 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 'object_number=7, tag_tid=1, entry_tid=6783' = '' 2024-09-12T06:48:57.496 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ last_mirror_pos='object_number=7, tag_tid=1, entry_tid=6783' 2024-09-12T06:48:57.496 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-09-12T06:48:57.496 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-12T06:48:57.496 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:48:57.497 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_tag=2 2024-09-12T06:48:57.497 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=7, tag_tid=1, entry_tid=6783' 2024-09-12T06:48:57.498 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-12T06:48:57.498 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:48:57.499 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_tag=1 2024-09-12T06:48:57.499 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-09-12T06:48:57.499 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-12T06:48:57.499 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:48:57.500 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_entry=1 2024-09-12T06:48:57.501 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=7, tag_tid=1, entry_tid=6783' 2024-09-12T06:48:57.501 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-12T06:48:57.501 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:48:57.502 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_entry=6783 2024-09-12T06:48:57.502 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 2 = 1 -a 1 -le 6783 2024-09-12T06:48:57.502 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:48:57.502 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:48:57.502 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:48:57.704 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror image_27 2024-09-12T06:48:57.704 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:48:57.705 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:57.705 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_27 2024-09-12T06:48:57.705 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:48:57.705 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n image_27 ']' 2024-09-12T06:48:57.705 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/image_27' 2024-09-12T06:48:57.705 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_27 2024-09-12T06:48:57.705 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:48:57.705 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:48:57.705 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:48:57.705 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:48:57.705 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:48:57.706 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:48:57.706 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:48:57.706 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:48:57.706 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:48:57.709 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:48:57.709 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_27 2024-09-12T06:48:57.709 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:48:57.709 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:48:57.709 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:48:57.709 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:48:57.709 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:48:57.709 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:48:57.709 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:48:57.709 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:48:57.709 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:48:57.710 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:48:57.710 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:48:57.710 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:48:57.710 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:48:57.710 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:48:57.710 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:48:57.710 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:48:57.710 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:48:57.710 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:48:57.710 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:48:57.710 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:48:57.710 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:48:57.711 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:48:57.711 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:48:57.711 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:48:57.711 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:48:57.711 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:48:57.711 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:48:57.711 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:48:57.711 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:48:57.711 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:48:57.725 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:48:57.725 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:48:57.725 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:48:57.726 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_27 2024-09-12T06:48:57.831 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:48:57.831 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror image_27 2024-09-12T06:48:57.831 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:48:57.831 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:48:57.831 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_27 2024-09-12T06:48:57.831 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_27 '' 2024-09-12T06:48:57.831 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:48:57.831 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:48:57.831 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_27 2024-09-12T06:48:57.832 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:48:57.832 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_27.status 2024-09-12T06:48:57.832 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_27.status 2024-09-12T06:48:57.832 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:48:57.832 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_27.status 2024-09-12T06:48:57.833 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_27 2024-09-12T06:48:57.833 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_27.status 2024-09-12T06:48:57.878 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 1 2024-09-12T06:48:57.878 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 2 2024-09-12T06:48:57.878 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:48:57.879 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6783], [object_number=6, tag_tid=1, entry_tid=6782]]], state=connected] 2024-09-12T06:48:57.879 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6783], [object_number=6, tag_tid=1, entry_tid=6782]]], state=connected] 2024-09-12T06:48:57.881 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_27.status 2024-09-12T06:48:57.882 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-09-12T06:48:57.883 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror image_27 2024-09-12T06:48:57.883 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:48:57.884 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:48:57.884 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_27 2024-09-12T06:48:57.884 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_27 '..*' 2024-09-12T06:48:57.884 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:48:57.884 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:48:57.884 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_27 2024-09-12T06:48:57.884 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:48:57.884 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_27.status 2024-09-12T06:48:57.884 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_27.status 2024-09-12T06:48:57.884 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:48:57.885 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_27.status 2024-09-12T06:48:57.885 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_27 2024-09-12T06:48:57.885 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_27.status 2024-09-12T06:48:57.931 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 1 2024-09-12T06:48:57.931 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 2 2024-09-12T06:48:57.931 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:48:57.931 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6783], [object_number=6, tag_tid=1, entry_tid=6782]]], state=connected] 2024-09-12T06:48:57.931 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6783], [object_number=6, tag_tid=1, entry_tid=6782]]], state=connected] 2024-09-12T06:48:57.934 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_27.status 2024-09-12T06:48:57.936 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-09-12T06:48:57.937 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-09-12T06:48:57.937 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:48:57.937 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_snap_present cluster1 mirror image_27 snap 2024-09-12T06:48:57.937 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:48:57.937 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:57.937 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_27 2024-09-12T06:48:57.937 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:48:57.937 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:48:57.937 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T06:48:57.937 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:48:58.937 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 info mirror/image_27@snap 2024-09-12T06:48:58.992 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:rbd image 'image_27': 2024-09-12T06:48:58.992 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: size 128 MiB in 32 objects 2024-09-12T06:48:58.992 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: order 22 (4 MiB objects) 2024-09-12T06:48:58.992 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: snapshot_count: 1 2024-09-12T06:48:58.992 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: id: 10a8fd50ea0 2024-09-12T06:48:58.992 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: block_name_prefix: rbd_data.10a8fd50ea0 2024-09-12T06:48:58.992 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: format: 2 2024-09-12T06:48:58.993 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: features: layering, exclusive-lock, journaling 2024-09-12T06:48:58.993 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: op_features: 2024-09-12T06:48:58.993 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: flags: 2024-09-12T06:48:58.993 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: create_timestamp: Thu Sep 12 06:42:24 2024 2024-09-12T06:48:58.993 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: access_timestamp: Thu Sep 12 06:42:24 2024 2024-09-12T06:48:58.993 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: modify_timestamp: Thu Sep 12 06:42:24 2024 2024-09-12T06:48:58.993 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: protected: False 2024-09-12T06:48:58.993 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: journal: 10a8fd50ea0 2024-09-12T06:48:58.993 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring state: enabled 2024-09-12T06:48:58.993 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring mode: journal 2024-09-12T06:48:58.993 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring global id: e8f8cb6c-fc38-4d9a-92ac-4a9cdc36dbdf 2024-09-12T06:48:58.993 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring primary: false 2024-09-12T06:48:58.996 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:48:58.996 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror image_27 snap 2024-09-12T06:48:58.997 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:58.997 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_27 2024-09-12T06:48:58.997 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:48:58.997 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:48:58.997 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_27.export 2024-09-12T06:48:58.997 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_27.export 2024-09-12T06:48:58.997 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_27.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_27.export 2024-09-12T06:48:58.997 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export image_27@snap /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_27.export 2024-09-12T06:48:59.154 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:48:59.160 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export image_27@snap /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_27.export 2024-09-12T06:48:59.356 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:48:59.362 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_27.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_27.export 2024-09-12T06:48:59.416 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_27.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_27.export 2024-09-12T06:48:59.449 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:48:59.450 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:48:59.450 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_28 2024-09-12T06:48:59.450 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_snap cluster2 mirror image_28 snap 2024-09-12T06:48:59.450 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:48:59.450 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:59.450 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_28 2024-09-12T06:48:59.450 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:48:59.450 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror snap create image_28@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T06:48:59.957 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_started cluster1 mirror image_28 2024-09-12T06:48:59.958 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:48:59.958 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:59.958 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_28 2024-09-12T06:48:59.958 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_state cluster1 mirror image_28 started 2024-09-12T06:48:59.958 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:48:59.958 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:48:59.958 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_28 2024-09-12T06:48:59.958 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=started 2024-09-12T06:48:59.958 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:48:59.958 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:48:59.958 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:49:00.959 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_replay_state cluster1 mirror image_28 started 2024-09-12T06:49:00.960 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:49:00.960 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:00.960 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_28 2024-09-12T06:49:00.960 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=started 2024-09-12T06:49:00.960 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local status_result 2024-09-12T06:49:00.960 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=stopped 2024-09-12T06:49:00.960 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_28 2024-09-12T06:49:00.960 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_instance=cluster1 2024-09-12T06:49:00.960 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -i state 2024-09-12T06:49:00.960 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:49:00.961 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:49:00.961 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance=cluster1 2024-09-12T06:49:00.961 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local loop_instance 2024-09-12T06:49:00.961 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:49:00.961 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sleep 0 2024-09-12T06:49:00.961 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:49:00.967 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ seq 0 1 2024-09-12T06:49:00.967 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:49:00.967 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_28 2024-09-12T06:49:00.967 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:0 2024-09-12T06:49:00.967 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:49:00.967 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:49:00.968 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:49:00.968 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:49:00.968 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:49:00.968 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:49:00.968 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:49:00.969 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:49:00.969 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:49:00.969 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:49:00.969 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:49:00.969 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:49:00.969 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:49:00.969 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:49:00.969 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:0 2024-09-12T06:49:00.969 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:49:00.969 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:49:00.969 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:49:00.969 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:0 2024-09-12T06:49:00.969 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:49:00.969 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:49:00.969 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:49:00.970 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:49:00.970 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T06:49:00.970 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:49:00.970 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:49:00.970 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=0 2024-09-12T06:49:00.970 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=0 2024-09-12T06:49:00.970 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:49:00.982 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:00.982 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:00.982 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:00.983 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_28 2024-09-12T06:49:01.077 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2024-09-12T06:49:01.077 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T06:49:01.077 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:admin_socket: invalid command 2024-09-12T06:49:01.080 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:49:01.080 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_28 2024-09-12T06:49:01.080 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:1 2024-09-12T06:49:01.080 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:49:01.080 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:49:01.081 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T06:49:01.081 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:1 2024-09-12T06:49:01.081 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:49:01.081 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:49:01.081 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:49:01.081 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:49:01.081 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T06:49:01.081 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:49:01.081 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:49:01.081 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=1 2024-09-12T06:49:01.081 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:49:01.081 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T06:49:01.081 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:1 2024-09-12T06:49:01.081 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:49:01.082 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:49:01.082 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T06:49:01.082 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:1 2024-09-12T06:49:01.082 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:49:01.082 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:49:01.082 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:49:01.082 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:49:01.082 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T06:49:01.082 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:49:01.082 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:49:01.082 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=1 2024-09-12T06:49:01.082 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=1 2024-09-12T06:49:01.082 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T06:49:01.101 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:49:01.101 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:49:01.101 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:49:01.101 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_28 2024-09-12T06:49:01.201 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ return 0 2024-09-12T06:49:01.202 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T06:49:01.202 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo ' "state": "Replaying"' 2024-09-12T06:49:01.202 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep -i Replaying 2024-09-12T06:49:01.203 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: "state": "Replaying" 2024-09-12T06:49:01.204 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_state=started 2024-09-12T06:49:01.204 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test started = started 2024-09-12T06:49:01.205 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:49:01.205 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_28 2024-09-12T06:49:01.205 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:49:01.205 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:49:01.205 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:49:01.205 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:49:01.205 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_28 2024-09-12T06:49:01.205 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' journal = journal ']' 2024-09-12T06:49:01.205 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror image_28 2024-09-12T06:49:01.205 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:49:01.205 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:49:01.205 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:49:01.205 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:49:01.205 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_28 2024-09-12T06:49:01.206 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-12T06:49:01.206 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-12T06:49:01.206 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:49:01.206 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:49:01.206 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:49:01.405 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror image_28 2024-09-12T06:49:01.405 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:49:01.405 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:01.405 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_28 2024-09-12T06:49:01.405 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:49:01.405 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n image_28 ']' 2024-09-12T06:49:01.405 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/image_28' 2024-09-12T06:49:01.405 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_28 2024-09-12T06:49:01.405 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:49:01.405 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:49:01.405 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:49:01.406 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:49:01.406 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:49:01.406 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:49:01.406 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:49:01.406 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:49:01.406 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:49:01.409 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:49:01.409 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_28 2024-09-12T06:49:01.409 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:49:01.409 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:49:01.409 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:49:01.409 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:49:01.409 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:49:01.410 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:49:01.410 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:49:01.410 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:49:01.410 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:49:01.410 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:49:01.410 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:49:01.410 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:49:01.410 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:49:01.410 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:49:01.410 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:49:01.410 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:49:01.410 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:49:01.410 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:49:01.411 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:49:01.411 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:49:01.411 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:49:01.411 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:49:01.411 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:49:01.411 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:49:01.411 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:49:01.411 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:49:01.411 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:49:01.411 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:49:01.411 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:49:01.411 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:49:01.427 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:01.427 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:01.427 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:01.427 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_28 2024-09-12T06:49:01.513 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2024-09-12T06:49:01.513 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd mirror flush : flush rbd mirror 2024-09-12T06:49:01.513 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:admin_socket: invalid command 2024-09-12T06:49:01.516 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:49:01.516 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_28 2024-09-12T06:49:01.516 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:1 2024-09-12T06:49:01.517 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:49:01.517 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:49:01.517 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-12T06:49:01.517 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:1 2024-09-12T06:49:01.517 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:49:01.517 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:49:01.517 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:49:01.517 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=1 2024-09-12T06:49:01.517 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-12T06:49:01.517 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:49:01.517 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:49:01.517 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=1 2024-09-12T06:49:01.517 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:49:01.518 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-12T06:49:01.518 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:1 2024-09-12T06:49:01.518 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:49:01.518 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:49:01.518 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T06:49:01.518 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:1 2024-09-12T06:49:01.518 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:49:01.518 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:49:01.518 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:49:01.518 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:49:01.518 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T06:49:01.518 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:49:01.518 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:49:01.519 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=1 2024-09-12T06:49:01.519 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:49:01.519 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T06:49:01.536 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:49:01.536 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:49:01.536 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:49:01.536 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_28 2024-09-12T06:49:01.633 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:49:01.633 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror image_28 2024-09-12T06:49:01.633 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:49:01.633 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:49:01.633 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_28 2024-09-12T06:49:01.633 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_28 '' 2024-09-12T06:49:01.633 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:49:01.634 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:49:01.634 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_28 2024-09-12T06:49:01.634 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:49:01.634 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_28.status 2024-09-12T06:49:01.634 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_28.status 2024-09-12T06:49:01.634 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:49:01.634 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_28.status 2024-09-12T06:49:01.635 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_28 2024-09-12T06:49:01.635 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_28.status 2024-09-12T06:49:01.679 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 1 2024-09-12T06:49:01.680 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 1 2024-09-12T06:49:01.680 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:49:01.680 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4695], [object_number=6, tag_tid=1, entry_tid=4694]]], state=connected] 2024-09-12T06:49:01.680 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4695], [object_number=6, tag_tid=1, entry_tid=4694]]], state=connected] 2024-09-12T06:49:01.682 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_28.status 2024-09-12T06:49:01.684 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-12T06:49:01.685 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror image_28 2024-09-12T06:49:01.685 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:49:01.685 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:49:01.685 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_28 2024-09-12T06:49:01.686 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_28 '..*' 2024-09-12T06:49:01.686 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:49:01.686 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:49:01.686 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_28 2024-09-12T06:49:01.686 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:49:01.686 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_28.status 2024-09-12T06:49:01.686 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_28.status 2024-09-12T06:49:01.686 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:49:01.687 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_28.status 2024-09-12T06:49:01.687 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_28 2024-09-12T06:49:01.687 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_28.status 2024-09-12T06:49:01.731 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 1 2024-09-12T06:49:01.731 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 1 2024-09-12T06:49:01.732 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:49:01.732 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4695], [object_number=6, tag_tid=1, entry_tid=4694]]], state=connected] 2024-09-12T06:49:01.732 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4695], [object_number=6, tag_tid=1, entry_tid=4694]]], state=connected] 2024-09-12T06:49:01.733 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_28.status 2024-09-12T06:49:01.736 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-12T06:49:01.736 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-09-12T06:49:01.736 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:49:01.736 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_snap_present cluster1 mirror image_28 snap 2024-09-12T06:49:01.736 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:49:01.736 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:01.736 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_28 2024-09-12T06:49:01.736 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:49:01.736 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:49:01.736 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T06:49:01.737 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:49:02.736 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 info mirror/image_28@snap 2024-09-12T06:49:02.784 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:rbd image 'image_28': 2024-09-12T06:49:02.785 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: size 128 MiB in 32 objects 2024-09-12T06:49:02.785 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: order 22 (4 MiB objects) 2024-09-12T06:49:02.785 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: snapshot_count: 1 2024-09-12T06:49:02.785 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: id: 10af3590f099 2024-09-12T06:49:02.785 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: block_name_prefix: rbd_data.10af3590f099 2024-09-12T06:49:02.785 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: format: 2 2024-09-12T06:49:02.785 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: features: layering, exclusive-lock, journaling 2024-09-12T06:49:02.785 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: op_features: 2024-09-12T06:49:02.785 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: flags: 2024-09-12T06:49:02.785 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: create_timestamp: Thu Sep 12 06:42:32 2024 2024-09-12T06:49:02.785 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: access_timestamp: Thu Sep 12 06:42:32 2024 2024-09-12T06:49:02.785 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: modify_timestamp: Thu Sep 12 06:43:33 2024 2024-09-12T06:49:02.785 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: protected: False 2024-09-12T06:49:02.786 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: journal: 10af3590f099 2024-09-12T06:49:02.786 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring state: enabled 2024-09-12T06:49:02.786 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring mode: journal 2024-09-12T06:49:02.786 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring global id: eb17bce3-fdc3-468d-b29e-3cd47301202d 2024-09-12T06:49:02.786 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring primary: false 2024-09-12T06:49:02.788 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:49:02.788 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror image_28 snap 2024-09-12T06:49:02.788 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:02.788 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_28 2024-09-12T06:49:02.788 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:49:02.788 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:49:02.788 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_28.export 2024-09-12T06:49:02.788 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_28.export 2024-09-12T06:49:02.789 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_28.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_28.export 2024-09-12T06:49:02.789 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export image_28@snap /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_28.export 2024-09-12T06:49:02.937 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:49:02.944 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export image_28@snap /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_28.export 2024-09-12T06:49:03.163 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:49:03.169 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_28.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_28.export 2024-09-12T06:49:03.224 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_28.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_28.export 2024-09-12T06:49:03.256 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:49:03.256 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:49:03.256 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_29 2024-09-12T06:49:03.256 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_snap cluster2 mirror image_29 snap 2024-09-12T06:49:03.256 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:49:03.256 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:03.256 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_29 2024-09-12T06:49:03.256 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:49:03.256 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror snap create image_29@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T06:49:04.978 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_started cluster1 mirror image_29 2024-09-12T06:49:04.978 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:49:04.978 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:04.978 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_29 2024-09-12T06:49:04.978 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_state cluster1 mirror image_29 started 2024-09-12T06:49:04.979 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:49:04.979 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:04.979 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_29 2024-09-12T06:49:04.979 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=started 2024-09-12T06:49:04.979 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:49:04.979 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:49:04.979 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:49:05.980 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_replay_state cluster1 mirror image_29 started 2024-09-12T06:49:05.980 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:49:05.980 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:05.981 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_29 2024-09-12T06:49:05.981 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=started 2024-09-12T06:49:05.981 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local status_result 2024-09-12T06:49:05.981 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=stopped 2024-09-12T06:49:05.981 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_29 2024-09-12T06:49:05.981 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_instance=cluster1 2024-09-12T06:49:05.981 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:49:05.981 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -i state 2024-09-12T06:49:05.981 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:49:05.981 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance=cluster1 2024-09-12T06:49:05.981 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local loop_instance 2024-09-12T06:49:05.981 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:49:05.982 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sleep 0 2024-09-12T06:49:05.985 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:49:05.985 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ seq 0 1 2024-09-12T06:49:05.985 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:49:05.985 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_29 2024-09-12T06:49:05.985 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:0 2024-09-12T06:49:05.985 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:49:05.985 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:49:05.985 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:49:05.985 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:49:05.986 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:49:05.986 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:49:05.986 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:49:05.986 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:49:05.986 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:49:05.986 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:49:05.986 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:49:05.986 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:49:05.986 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:49:05.986 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:49:05.986 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:0 2024-09-12T06:49:05.986 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:49:05.986 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:49:05.987 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:49:05.987 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:0 2024-09-12T06:49:05.987 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:49:05.987 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:49:05.987 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:49:05.987 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:49:05.987 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T06:49:05.987 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:49:05.987 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:49:05.987 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=0 2024-09-12T06:49:05.987 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=0 2024-09-12T06:49:05.987 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:49:06.003 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:06.003 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:06.003 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:06.004 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_29 2024-09-12T06:49:06.103 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ return 0 2024-09-12T06:49:06.104 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T06:49:06.104 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo ' "state": "Replaying"' 2024-09-12T06:49:06.104 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep -i Replaying 2024-09-12T06:49:06.105 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: "state": "Replaying" 2024-09-12T06:49:06.107 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_state=started 2024-09-12T06:49:06.107 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test started = started 2024-09-12T06:49:06.107 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:49:06.107 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_29 2024-09-12T06:49:06.107 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:49:06.107 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:49:06.107 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:49:06.107 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:49:06.107 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_29 2024-09-12T06:49:06.107 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' journal = journal ']' 2024-09-12T06:49:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror image_29 2024-09-12T06:49:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:49:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:49:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:49:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:49:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_29 2024-09-12T06:49:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-12T06:49:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-12T06:49:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:49:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:49:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:49:06.308 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror image_29 2024-09-12T06:49:06.308 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:49:06.308 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:06.308 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_29 2024-09-12T06:49:06.308 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:49:06.309 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n image_29 ']' 2024-09-12T06:49:06.309 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/image_29' 2024-09-12T06:49:06.309 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_29 2024-09-12T06:49:06.309 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:49:06.309 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:49:06.309 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:49:06.309 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:49:06.309 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:49:06.309 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:49:06.309 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:49:06.309 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:49:06.309 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:49:06.313 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:49:06.313 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_29 2024-09-12T06:49:06.313 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:49:06.313 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:49:06.313 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:49:06.313 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:49:06.313 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:49:06.313 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:49:06.313 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:49:06.313 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:49:06.313 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:49:06.313 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:49:06.314 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:49:06.314 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:49:06.314 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:49:06.314 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:49:06.314 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:49:06.314 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:49:06.314 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:49:06.314 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:49:06.314 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:49:06.314 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:49:06.314 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:49:06.314 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:49:06.314 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:49:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:49:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:49:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:49:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:49:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:49:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:49:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:49:06.329 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:06.329 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:06.330 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:06.330 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_29 2024-09-12T06:49:06.424 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:49:06.425 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror image_29 2024-09-12T06:49:06.425 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:49:06.425 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:49:06.425 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_29 2024-09-12T06:49:06.425 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_29 '' 2024-09-12T06:49:06.426 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:49:06.426 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:49:06.426 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_29 2024-09-12T06:49:06.426 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:49:06.426 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_29.status 2024-09-12T06:49:06.426 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_29.status 2024-09-12T06:49:06.426 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:49:06.426 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_29.status 2024-09-12T06:49:06.426 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_29 2024-09-12T06:49:06.427 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_29.status 2024-09-12T06:49:06.472 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 1 2024-09-12T06:49:06.472 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 1 2024-09-12T06:49:06.473 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:49:06.473 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5795], [object_number=6, tag_tid=1, entry_tid=5794]]], state=connected] 2024-09-12T06:49:06.473 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=5, tag_tid=1, entry_tid=5797], [object_number=4, tag_tid=1, entry_tid=5796], [object_number=7, tag_tid=1, entry_tid=5795], [object_number=6, tag_tid=1, entry_tid=5794]]], state=connected] 2024-09-12T06:49:06.475 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_29.status 2024-09-12T06:49:06.477 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-09-12T06:49:06.478 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror image_29 2024-09-12T06:49:06.478 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:49:06.478 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:49:06.478 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_29 2024-09-12T06:49:06.478 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_29 '..*' 2024-09-12T06:49:06.478 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:49:06.478 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:49:06.479 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_29 2024-09-12T06:49:06.479 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:49:06.479 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_29.status 2024-09-12T06:49:06.479 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_29.status 2024-09-12T06:49:06.479 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:49:06.479 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_29.status 2024-09-12T06:49:06.480 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_29 2024-09-12T06:49:06.480 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_29.status 2024-09-12T06:49:06.524 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 1 2024-09-12T06:49:06.524 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 1 2024-09-12T06:49:06.524 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:49:06.524 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5795], [object_number=6, tag_tid=1, entry_tid=5794]]], state=connected] 2024-09-12T06:49:06.524 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=5, tag_tid=1, entry_tid=5797], [object_number=4, tag_tid=1, entry_tid=5796], [object_number=7, tag_tid=1, entry_tid=5795], [object_number=6, tag_tid=1, entry_tid=5794]]], state=connected] 2024-09-12T06:49:06.527 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_29.status 2024-09-12T06:49:06.529 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=5, tag_tid=1, entry_tid=5797' 2024-09-12T06:49:06.530 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=5, tag_tid=1, entry_tid=5797' 2024-09-12T06:49:06.530 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 'object_number=5, tag_tid=1, entry_tid=5797' '!=' '' 2024-09-12T06:49:06.530 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ break 2024-09-12T06:49:06.530 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 'object_number=5, tag_tid=1, entry_tid=5797' = '' 2024-09-12T06:49:06.530 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ last_mirror_pos='object_number=5, tag_tid=1, entry_tid=5797' 2024-09-12T06:49:06.530 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-09-12T06:49:06.530 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-12T06:49:06.530 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:49:06.531 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_tag=2 2024-09-12T06:49:06.531 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=5, tag_tid=1, entry_tid=5797' 2024-09-12T06:49:06.531 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-12T06:49:06.532 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:49:06.532 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_tag=1 2024-09-12T06:49:06.533 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-09-12T06:49:06.533 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-12T06:49:06.533 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:49:06.534 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_entry=1 2024-09-12T06:49:06.534 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=5, tag_tid=1, entry_tid=5797' 2024-09-12T06:49:06.534 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-12T06:49:06.534 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:49:06.535 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_entry=5797 2024-09-12T06:49:06.536 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 2 = 1 -a 1 -le 5797 2024-09-12T06:49:06.536 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:49:06.536 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:49:06.536 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:49:06.738 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror image_29 2024-09-12T06:49:06.738 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:49:06.738 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:06.738 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_29 2024-09-12T06:49:06.738 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:49:06.738 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n image_29 ']' 2024-09-12T06:49:06.738 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/image_29' 2024-09-12T06:49:06.738 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_29 2024-09-12T06:49:06.739 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:49:06.739 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:49:06.739 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:49:06.739 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:49:06.739 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:49:06.739 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:49:06.739 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:49:06.739 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:49:06.739 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:49:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:49:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_29 2024-09-12T06:49:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:49:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:49:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:49:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:49:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:49:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:49:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:49:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:49:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:49:06.744 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:49:06.744 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:49:06.744 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:49:06.744 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:49:06.744 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:49:06.744 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:49:06.744 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:49:06.744 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:49:06.744 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:49:06.744 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:49:06.744 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:49:06.744 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:49:06.744 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:49:06.745 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:49:06.745 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:49:06.745 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:49:06.745 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:49:06.745 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:49:06.745 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:49:06.745 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:49:06.745 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:49:06.759 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:06.759 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:06.759 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:06.759 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_29 2024-09-12T06:49:06.857 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:49:06.857 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror image_29 2024-09-12T06:49:06.858 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:49:06.858 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:49:06.858 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_29 2024-09-12T06:49:06.858 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_29 '' 2024-09-12T06:49:06.858 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:49:06.858 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:49:06.858 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_29 2024-09-12T06:49:06.858 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:49:06.858 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_29.status 2024-09-12T06:49:06.858 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_29.status 2024-09-12T06:49:06.858 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:49:06.859 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_29.status 2024-09-12T06:49:06.859 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_29 2024-09-12T06:49:06.859 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_29.status 2024-09-12T06:49:06.905 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 1 2024-09-12T06:49:06.905 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 1 2024-09-12T06:49:06.905 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:49:06.905 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5795], [object_number=6, tag_tid=1, entry_tid=5794]]], state=connected] 2024-09-12T06:49:06.905 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5795], [object_number=6, tag_tid=1, entry_tid=5794]]], state=connected] 2024-09-12T06:49:06.907 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_29.status 2024-09-12T06:49:06.910 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-09-12T06:49:06.910 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror image_29 2024-09-12T06:49:06.910 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:49:06.911 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:49:06.911 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_29 2024-09-12T06:49:06.911 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_29 '..*' 2024-09-12T06:49:06.911 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:49:06.911 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:49:06.911 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_29 2024-09-12T06:49:06.911 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:49:06.911 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_29.status 2024-09-12T06:49:06.911 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_29.status 2024-09-12T06:49:06.911 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:49:06.912 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_29.status 2024-09-12T06:49:06.912 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_29 2024-09-12T06:49:06.912 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_29.status 2024-09-12T06:49:06.957 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 1 2024-09-12T06:49:06.957 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 1 2024-09-12T06:49:06.957 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:49:06.958 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5795], [object_number=6, tag_tid=1, entry_tid=5794]]], state=connected] 2024-09-12T06:49:06.958 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5795], [object_number=6, tag_tid=1, entry_tid=5794]]], state=connected] 2024-09-12T06:49:06.960 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_29.status 2024-09-12T06:49:06.962 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-09-12T06:49:06.963 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-09-12T06:49:06.963 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:49:06.963 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_snap_present cluster1 mirror image_29 snap 2024-09-12T06:49:06.963 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:49:06.963 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:06.963 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_29 2024-09-12T06:49:06.963 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:49:06.963 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:49:06.963 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T06:49:06.963 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:49:07.963 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 info mirror/image_29@snap 2024-09-12T06:49:08.012 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:rbd image 'image_29': 2024-09-12T06:49:08.012 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: size 128 MiB in 32 objects 2024-09-12T06:49:08.012 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: order 22 (4 MiB objects) 2024-09-12T06:49:08.012 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: snapshot_count: 1 2024-09-12T06:49:08.012 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: id: 10a8d69b8414 2024-09-12T06:49:08.012 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: block_name_prefix: rbd_data.10a8d69b8414 2024-09-12T06:49:08.012 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: format: 2 2024-09-12T06:49:08.012 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: features: layering, exclusive-lock, journaling 2024-09-12T06:49:08.013 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: op_features: 2024-09-12T06:49:08.013 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: flags: 2024-09-12T06:49:08.013 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: create_timestamp: Thu Sep 12 06:42:37 2024 2024-09-12T06:49:08.013 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: access_timestamp: Thu Sep 12 06:42:37 2024 2024-09-12T06:49:08.013 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: modify_timestamp: Thu Sep 12 06:42:37 2024 2024-09-12T06:49:08.013 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: protected: False 2024-09-12T06:49:08.013 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: journal: 10a8d69b8414 2024-09-12T06:49:08.013 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring state: enabled 2024-09-12T06:49:08.013 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring mode: journal 2024-09-12T06:49:08.013 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring global id: db8cd863-47ae-4ecd-b225-248836999365 2024-09-12T06:49:08.013 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring primary: false 2024-09-12T06:49:08.016 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:49:08.016 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror image_29 snap 2024-09-12T06:49:08.016 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:08.016 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_29 2024-09-12T06:49:08.016 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:49:08.016 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:49:08.016 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_29.export 2024-09-12T06:49:08.016 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_29.export 2024-09-12T06:49:08.016 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_29.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_29.export 2024-09-12T06:49:08.017 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export image_29@snap /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_29.export 2024-09-12T06:49:08.160 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:49:08.167 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export image_29@snap /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_29.export 2024-09-12T06:49:08.341 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:49:08.347 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_29.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_29.export 2024-09-12T06:49:08.402 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_29.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_29.export 2024-09-12T06:49:08.433 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:49:08.433 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:49:08.434 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_30 2024-09-12T06:49:08.434 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_snap cluster2 mirror image_30 snap 2024-09-12T06:49:08.434 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:49:08.434 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:08.434 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_30 2024-09-12T06:49:08.434 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:49:08.434 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror snap create image_30@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T06:49:09.991 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_started cluster1 mirror image_30 2024-09-12T06:49:09.991 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:49:09.991 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:09.991 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_30 2024-09-12T06:49:09.991 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_state cluster1 mirror image_30 started 2024-09-12T06:49:09.991 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:49:09.991 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:09.991 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_30 2024-09-12T06:49:09.991 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=started 2024-09-12T06:49:09.992 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:49:09.992 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:49:09.992 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:49:10.991 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_replay_state cluster1 mirror image_30 started 2024-09-12T06:49:10.991 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:49:10.992 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:10.992 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_30 2024-09-12T06:49:10.992 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=started 2024-09-12T06:49:10.992 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local status_result 2024-09-12T06:49:10.992 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=stopped 2024-09-12T06:49:10.992 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_30 2024-09-12T06:49:10.998 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -i state 2024-09-12T06:49:10.998 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_instance=cluster1 2024-09-12T06:49:10.998 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:49:10.998 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:49:10.998 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance=cluster1 2024-09-12T06:49:10.998 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local loop_instance 2024-09-12T06:49:10.998 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:49:10.998 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sleep 0 2024-09-12T06:49:10.998 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:49:10.998 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ seq 0 1 2024-09-12T06:49:10.998 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:49:10.999 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_30 2024-09-12T06:49:10.999 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:0 2024-09-12T06:49:10.999 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:49:10.999 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:49:10.999 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:49:10.999 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:49:10.999 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:49:10.999 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:49:10.999 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:49:10.999 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:49:10.999 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:49:10.999 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:49:10.999 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:49:10.999 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:49:11.000 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:49:11.000 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:49:11.000 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:0 2024-09-12T06:49:11.000 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:49:11.000 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:49:11.000 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:49:11.000 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:0 2024-09-12T06:49:11.000 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:49:11.000 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:49:11.000 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:49:11.000 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:49:11.000 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T06:49:11.000 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:49:11.001 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:49:11.001 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=0 2024-09-12T06:49:11.001 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=0 2024-09-12T06:49:11.001 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:49:11.018 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:11.018 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:11.018 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:11.018 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_30 2024-09-12T06:49:11.118 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2024-09-12T06:49:11.118 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T06:49:11.118 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:admin_socket: invalid command 2024-09-12T06:49:11.122 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:49:11.123 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_30 2024-09-12T06:49:11.123 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:1 2024-09-12T06:49:11.123 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:49:11.123 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:49:11.123 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T06:49:11.123 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:1 2024-09-12T06:49:11.124 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:49:11.124 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:49:11.124 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:49:11.124 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:49:11.124 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T06:49:11.124 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:49:11.124 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:49:11.124 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=1 2024-09-12T06:49:11.124 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:49:11.124 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T06:49:11.124 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:1 2024-09-12T06:49:11.124 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:49:11.124 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:49:11.125 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T06:49:11.125 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:1 2024-09-12T06:49:11.125 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:49:11.125 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:49:11.125 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:49:11.125 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:49:11.125 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T06:49:11.125 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:49:11.125 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:49:11.125 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=1 2024-09-12T06:49:11.126 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=1 2024-09-12T06:49:11.126 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T06:49:11.141 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:49:11.141 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:49:11.141 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:49:11.141 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_30 2024-09-12T06:49:11.242 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ return 0 2024-09-12T06:49:11.242 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T06:49:11.243 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo ' "state": "Replaying"' 2024-09-12T06:49:11.243 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep -i Replaying 2024-09-12T06:49:11.243 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: "state": "Replaying" 2024-09-12T06:49:11.245 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_state=started 2024-09-12T06:49:11.245 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test started = started 2024-09-12T06:49:11.245 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:49:11.246 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_30 2024-09-12T06:49:11.246 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:49:11.246 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:49:11.246 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:49:11.246 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:49:11.246 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_30 2024-09-12T06:49:11.246 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' journal = journal ']' 2024-09-12T06:49:11.246 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror image_30 2024-09-12T06:49:11.246 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:49:11.246 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:49:11.246 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:49:11.246 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:49:11.246 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_30 2024-09-12T06:49:11.247 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-12T06:49:11.247 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-12T06:49:11.247 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:49:11.247 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:49:11.247 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:49:11.446 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror image_30 2024-09-12T06:49:11.446 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:49:11.446 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:11.446 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_30 2024-09-12T06:49:11.447 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:49:11.447 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n image_30 ']' 2024-09-12T06:49:11.447 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/image_30' 2024-09-12T06:49:11.447 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_30 2024-09-12T06:49:11.447 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:49:11.447 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:49:11.447 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:49:11.447 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:49:11.447 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:49:11.447 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:49:11.447 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:49:11.447 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:49:11.447 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:49:11.450 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:49:11.450 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_30 2024-09-12T06:49:11.450 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:49:11.450 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:49:11.450 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:49:11.451 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:49:11.451 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:49:11.451 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:49:11.451 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:49:11.451 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:49:11.451 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:49:11.451 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:49:11.451 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:49:11.451 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:49:11.451 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:49:11.451 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:49:11.451 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:49:11.451 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:49:11.451 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:49:11.452 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:49:11.452 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:49:11.452 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:49:11.452 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:49:11.452 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:49:11.452 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:49:11.452 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:49:11.452 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:49:11.452 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:49:11.452 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:49:11.452 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:49:11.452 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:49:11.452 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:49:11.468 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:11.468 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:11.468 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:11.468 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_30 2024-09-12T06:49:11.554 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2024-09-12T06:49:11.554 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd mirror flush : flush rbd mirror 2024-09-12T06:49:11.554 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:admin_socket: invalid command 2024-09-12T06:49:11.558 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:49:11.559 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_30 2024-09-12T06:49:11.559 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:1 2024-09-12T06:49:11.559 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:49:11.559 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:49:11.559 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-12T06:49:11.559 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:1 2024-09-12T06:49:11.559 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:49:11.559 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:49:11.559 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:49:11.559 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=1 2024-09-12T06:49:11.559 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-12T06:49:11.559 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:49:11.559 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:49:11.560 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=1 2024-09-12T06:49:11.560 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:49:11.560 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-12T06:49:11.560 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:1 2024-09-12T06:49:11.560 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:49:11.560 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:49:11.560 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T06:49:11.560 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:1 2024-09-12T06:49:11.560 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:49:11.560 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:49:11.560 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:49:11.561 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:49:11.561 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T06:49:11.561 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:49:11.561 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:49:11.561 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=1 2024-09-12T06:49:11.561 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:49:11.561 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T06:49:11.575 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:49:11.575 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:49:11.575 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:49:11.575 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_30 2024-09-12T06:49:11.675 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:49:11.675 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror image_30 2024-09-12T06:49:11.676 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:49:11.676 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:49:11.676 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_30 2024-09-12T06:49:11.676 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_30 '' 2024-09-12T06:49:11.676 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:49:11.676 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:49:11.677 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_30 2024-09-12T06:49:11.677 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:49:11.677 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_30.status 2024-09-12T06:49:11.677 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_30.status 2024-09-12T06:49:11.677 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:49:11.677 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_30.status 2024-09-12T06:49:11.677 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_30 2024-09-12T06:49:11.678 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_30.status 2024-09-12T06:49:11.719 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 1 2024-09-12T06:49:11.720 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 1 2024-09-12T06:49:11.720 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:49:11.720 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4539], [object_number=6, tag_tid=1, entry_tid=4538]]], state=connected] 2024-09-12T06:49:11.720 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4539], [object_number=6, tag_tid=1, entry_tid=4538]]], state=connected] 2024-09-12T06:49:11.722 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_30.status 2024-09-12T06:49:11.724 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-09-12T06:49:11.725 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror image_30 2024-09-12T06:49:11.725 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:49:11.725 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:49:11.725 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_30 2024-09-12T06:49:11.725 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_30 '..*' 2024-09-12T06:49:11.725 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:49:11.725 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:49:11.726 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_30 2024-09-12T06:49:11.726 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:49:11.726 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_30.status 2024-09-12T06:49:11.726 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_30.status 2024-09-12T06:49:11.726 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:49:11.726 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_30.status 2024-09-12T06:49:11.726 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_30 2024-09-12T06:49:11.727 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_30.status 2024-09-12T06:49:11.771 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 1 2024-09-12T06:49:11.772 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 1 2024-09-12T06:49:11.772 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:49:11.772 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4539], [object_number=6, tag_tid=1, entry_tid=4538]]], state=connected] 2024-09-12T06:49:11.772 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4539], [object_number=6, tag_tid=1, entry_tid=4538]]], state=connected] 2024-09-12T06:49:11.774 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_30.status 2024-09-12T06:49:11.777 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-09-12T06:49:11.777 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-09-12T06:49:11.777 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:49:11.778 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_snap_present cluster1 mirror image_30 snap 2024-09-12T06:49:11.778 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:49:11.778 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:11.778 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_30 2024-09-12T06:49:11.778 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:49:11.778 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:49:11.778 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T06:49:11.778 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:49:12.777 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 info mirror/image_30@snap 2024-09-12T06:49:12.823 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:rbd image 'image_30': 2024-09-12T06:49:12.823 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: size 128 MiB in 32 objects 2024-09-12T06:49:12.823 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: order 22 (4 MiB objects) 2024-09-12T06:49:12.823 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: snapshot_count: 1 2024-09-12T06:49:12.823 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: id: 10af8fcfb97f 2024-09-12T06:49:12.824 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: block_name_prefix: rbd_data.10af8fcfb97f 2024-09-12T06:49:12.824 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: format: 2 2024-09-12T06:49:12.824 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: features: layering, exclusive-lock, journaling 2024-09-12T06:49:12.824 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: op_features: 2024-09-12T06:49:12.824 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: flags: 2024-09-12T06:49:12.824 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: create_timestamp: Thu Sep 12 06:42:44 2024 2024-09-12T06:49:12.824 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: access_timestamp: Thu Sep 12 06:42:44 2024 2024-09-12T06:49:12.824 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: modify_timestamp: Thu Sep 12 06:42:44 2024 2024-09-12T06:49:12.824 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: protected: False 2024-09-12T06:49:12.824 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: journal: 10af8fcfb97f 2024-09-12T06:49:12.824 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring state: enabled 2024-09-12T06:49:12.824 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring mode: journal 2024-09-12T06:49:12.824 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring global id: 5eb4a44b-e1e5-4490-98f1-dd782905ac40 2024-09-12T06:49:12.825 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring primary: false 2024-09-12T06:49:12.827 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:49:12.827 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror image_30 snap 2024-09-12T06:49:12.827 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:12.827 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_30 2024-09-12T06:49:12.828 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:49:12.828 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:49:12.828 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_30.export 2024-09-12T06:49:12.828 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_30.export 2024-09-12T06:49:12.828 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_30.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_30.export 2024-09-12T06:49:12.828 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export image_30@snap /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_30.export 2024-09-12T06:49:12.971 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:49:12.976 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export image_30@snap /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_30.export 2024-09-12T06:49:13.078 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:49:13.080+0000 7fac02620640 -1 bdev(0x55749a596c00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-09-12T06:49:13.078 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:49:13.080+0000 7fac0161e640 -1 bdev(0x55749a596c00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 2 2024-09-12T06:49:13.179 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:49:13.186 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_30.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_30.export 2024-09-12T06:49:13.242 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_30.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_30.export 2024-09-12T06:49:13.273 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:49:13.273 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:49:13.273 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_31 2024-09-12T06:49:13.274 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_snap cluster2 mirror image_31 snap 2024-09-12T06:49:13.274 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:49:13.274 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:13.274 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_31 2024-09-12T06:49:13.274 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:49:13.274 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror snap create image_31@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T06:49:14.013 INFO:tasks.ceph.cluster2.osd.1.smithi152.stderr:2024-09-12T06:49:14.014+0000 7f33e785a640 -1 bdev(0x55c73b90ec00 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 1 2024-09-12T06:49:14.013 INFO:tasks.ceph.cluster2.osd.1.smithi152.stderr:2024-09-12T06:49:14.014+0000 7f33e905d640 -1 bdev(0x55c73b90ec00 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 1 2024-09-12T06:49:15.029 INFO:tasks.ceph.cluster2.osd.0.smithi152.stderr:2024-09-12T06:49:15.027+0000 7fd61689f640 -1 reset not still connected to 0x55e764069790 2024-09-12T06:49:16.015 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_started cluster1 mirror image_31 2024-09-12T06:49:16.015 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:49:16.015 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:16.015 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_31 2024-09-12T06:49:16.016 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_state cluster1 mirror image_31 started 2024-09-12T06:49:16.016 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:49:16.016 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:16.016 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_31 2024-09-12T06:49:16.016 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=started 2024-09-12T06:49:16.016 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:49:16.016 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:49:16.016 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:49:17.017 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_replay_state cluster1 mirror image_31 started 2024-09-12T06:49:17.017 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:49:17.017 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:17.017 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_31 2024-09-12T06:49:17.017 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=started 2024-09-12T06:49:17.017 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local status_result 2024-09-12T06:49:17.017 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=stopped 2024-09-12T06:49:17.017 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_31 2024-09-12T06:49:17.017 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_instance=cluster1 2024-09-12T06:49:17.017 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -i state 2024-09-12T06:49:17.017 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:49:17.018 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:49:17.018 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance=cluster1 2024-09-12T06:49:17.018 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local loop_instance 2024-09-12T06:49:17.018 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:49:17.018 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sleep 0 2024-09-12T06:49:17.020 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:49:17.020 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ seq 0 1 2024-09-12T06:49:17.020 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:49:17.020 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_31 2024-09-12T06:49:17.020 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:0 2024-09-12T06:49:17.020 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:49:17.020 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:49:17.020 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:49:17.021 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:49:17.021 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:49:17.021 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:49:17.021 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:49:17.021 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:49:17.021 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:49:17.021 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:49:17.021 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:49:17.021 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:49:17.021 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:49:17.021 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:49:17.021 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:0 2024-09-12T06:49:17.023 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:49:17.023 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:49:17.023 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:49:17.023 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:0 2024-09-12T06:49:17.023 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:49:17.023 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:49:17.023 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:49:17.023 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:49:17.023 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T06:49:17.023 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:49:17.024 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:49:17.024 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=0 2024-09-12T06:49:17.024 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=0 2024-09-12T06:49:17.024 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:49:17.042 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:17.042 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:17.042 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:17.042 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_31 2024-09-12T06:49:17.151 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ return 0 2024-09-12T06:49:17.152 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T06:49:17.152 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo ' "state": "Replaying"' 2024-09-12T06:49:17.152 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep -i Replaying 2024-09-12T06:49:17.153 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: "state": "Replaying" 2024-09-12T06:49:17.155 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_state=started 2024-09-12T06:49:17.155 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test started = started 2024-09-12T06:49:17.155 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:49:17.155 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_31 2024-09-12T06:49:17.156 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:49:17.156 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:49:17.156 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:49:17.156 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:49:17.156 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_31 2024-09-12T06:49:17.156 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' journal = journal ']' 2024-09-12T06:49:17.156 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror image_31 2024-09-12T06:49:17.156 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:49:17.156 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:49:17.156 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:49:17.156 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:49:17.156 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_31 2024-09-12T06:49:17.156 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-12T06:49:17.157 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-12T06:49:17.157 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:49:17.157 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:49:17.157 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:49:17.357 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror image_31 2024-09-12T06:49:17.358 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:49:17.358 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:17.358 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_31 2024-09-12T06:49:17.358 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:49:17.358 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n image_31 ']' 2024-09-12T06:49:17.358 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/image_31' 2024-09-12T06:49:17.358 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_31 2024-09-12T06:49:17.358 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:49:17.358 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:49:17.358 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:49:17.358 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:49:17.358 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:49:17.359 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:49:17.359 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:49:17.359 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:49:17.359 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:49:17.362 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:49:17.362 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_31 2024-09-12T06:49:17.362 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:49:17.362 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:49:17.362 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:49:17.362 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:49:17.362 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:49:17.362 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:49:17.362 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:49:17.362 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:49:17.362 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:49:17.362 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:49:17.362 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:49:17.363 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:49:17.363 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:49:17.363 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:49:17.363 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:49:17.363 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:49:17.363 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:49:17.363 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:49:17.363 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:49:17.363 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:49:17.363 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:49:17.363 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:49:17.363 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:49:17.363 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:49:17.363 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:49:17.363 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:49:17.364 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:49:17.364 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:49:17.364 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:49:17.364 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:49:17.379 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:17.379 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:17.379 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:17.379 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_31 2024-09-12T06:49:17.481 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:49:17.481 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror image_31 2024-09-12T06:49:17.481 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:49:17.481 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:49:17.481 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_31 2024-09-12T06:49:17.481 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_31 '' 2024-09-12T06:49:17.481 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:49:17.481 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:49:17.482 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_31 2024-09-12T06:49:17.482 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:49:17.482 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_31.status 2024-09-12T06:49:17.482 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_31.status 2024-09-12T06:49:17.482 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:49:17.482 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_31.status 2024-09-12T06:49:17.483 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_31 2024-09-12T06:49:17.483 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_31.status 2024-09-12T06:49:17.527 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 2 2024-09-12T06:49:17.527 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 2 2024-09-12T06:49:17.527 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:49:17.527 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=7459], [object_number=10, tag_tid=1, entry_tid=7458]]], state=connected] 2024-09-12T06:49:17.527 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=7459], [object_number=10, tag_tid=1, entry_tid=7458]]], state=connected] 2024-09-12T06:49:17.530 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_31.status 2024-09-12T06:49:17.531 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-09-12T06:49:17.532 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror image_31 2024-09-12T06:49:17.533 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:49:17.533 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:49:17.533 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_31 2024-09-12T06:49:17.533 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_31 '..*' 2024-09-12T06:49:17.533 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:49:17.533 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:49:17.533 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_31 2024-09-12T06:49:17.533 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:49:17.533 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_31.status 2024-09-12T06:49:17.533 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_31.status 2024-09-12T06:49:17.533 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:49:17.534 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_31.status 2024-09-12T06:49:17.534 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_31 2024-09-12T06:49:17.534 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_31.status 2024-09-12T06:49:17.578 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 2 2024-09-12T06:49:17.579 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 2 2024-09-12T06:49:17.579 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:49:17.579 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=7459], [object_number=10, tag_tid=1, entry_tid=7458]]], state=connected] 2024-09-12T06:49:17.579 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=7459], [object_number=10, tag_tid=1, entry_tid=7458]]], state=connected] 2024-09-12T06:49:17.581 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_31.status 2024-09-12T06:49:17.584 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-09-12T06:49:17.584 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-09-12T06:49:17.584 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:49:17.584 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_snap_present cluster1 mirror image_31 snap 2024-09-12T06:49:17.584 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:49:17.584 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:17.584 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_31 2024-09-12T06:49:17.584 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:49:17.585 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:49:17.585 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T06:49:17.585 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:49:18.584 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 info mirror/image_31@snap 2024-09-12T06:49:18.632 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:rbd image 'image_31': 2024-09-12T06:49:18.633 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: size 128 MiB in 32 objects 2024-09-12T06:49:18.633 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: order 22 (4 MiB objects) 2024-09-12T06:49:18.633 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: snapshot_count: 1 2024-09-12T06:49:18.633 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: id: 10a823e48f95 2024-09-12T06:49:18.633 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: block_name_prefix: rbd_data.10a823e48f95 2024-09-12T06:49:18.633 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: format: 2 2024-09-12T06:49:18.633 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: features: layering, exclusive-lock, journaling 2024-09-12T06:49:18.633 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: op_features: 2024-09-12T06:49:18.633 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: flags: 2024-09-12T06:49:18.633 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: create_timestamp: Thu Sep 12 06:42:49 2024 2024-09-12T06:49:18.633 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: access_timestamp: Thu Sep 12 06:42:49 2024 2024-09-12T06:49:18.633 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: modify_timestamp: Thu Sep 12 06:43:50 2024 2024-09-12T06:49:18.633 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: protected: False 2024-09-12T06:49:18.634 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: journal: 10a823e48f95 2024-09-12T06:49:18.634 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring state: enabled 2024-09-12T06:49:18.634 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring mode: journal 2024-09-12T06:49:18.634 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring global id: 23e30da3-1912-4bdc-a55f-2fe20bbeec6d 2024-09-12T06:49:18.634 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring primary: false 2024-09-12T06:49:18.637 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:49:18.637 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror image_31 snap 2024-09-12T06:49:18.637 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:18.637 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_31 2024-09-12T06:49:18.637 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:49:18.637 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:49:18.637 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_31.export 2024-09-12T06:49:18.637 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_31.export 2024-09-12T06:49:18.637 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_31.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_31.export 2024-09-12T06:49:18.638 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export image_31@snap /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_31.export 2024-09-12T06:49:18.784 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:49:18.791 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export image_31@snap /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_31.export 2024-09-12T06:49:18.991 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:49:18.999 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_31.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_31.export 2024-09-12T06:49:19.053 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_31.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_31.export 2024-09-12T06:49:19.085 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:49:19.085 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:49:19.085 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_32 2024-09-12T06:49:19.085 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_snap cluster2 mirror image_32 snap 2024-09-12T06:49:19.086 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:49:19.086 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:19.086 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_32 2024-09-12T06:49:19.086 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:49:19.086 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror snap create image_32@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T06:49:21.027 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_started cluster1 mirror image_32 2024-09-12T06:49:21.028 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:49:21.028 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:21.028 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_32 2024-09-12T06:49:21.028 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_state cluster1 mirror image_32 started 2024-09-12T06:49:21.028 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:49:21.028 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:21.028 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_32 2024-09-12T06:49:21.029 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=started 2024-09-12T06:49:21.029 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:49:21.029 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:49:21.029 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:49:22.028 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_replay_state cluster1 mirror image_32 started 2024-09-12T06:49:22.029 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:49:22.029 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:22.029 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_32 2024-09-12T06:49:22.029 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=started 2024-09-12T06:49:22.029 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local status_result 2024-09-12T06:49:22.029 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=stopped 2024-09-12T06:49:22.029 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_32 2024-09-12T06:49:22.029 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -i state 2024-09-12T06:49:22.029 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_instance=cluster1 2024-09-12T06:49:22.030 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:49:22.030 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:49:22.030 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance=cluster1 2024-09-12T06:49:22.030 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local loop_instance 2024-09-12T06:49:22.030 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:49:22.030 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sleep 0 2024-09-12T06:49:22.031 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:49:22.031 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ seq 0 1 2024-09-12T06:49:22.032 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:49:22.032 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_32 2024-09-12T06:49:22.032 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:0 2024-09-12T06:49:22.032 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:49:22.032 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:49:22.032 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:49:22.032 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:49:22.032 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:49:22.032 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:49:22.032 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:49:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:49:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:49:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:49:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:49:22.035 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:49:22.035 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:49:22.035 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:49:22.035 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:0 2024-09-12T06:49:22.035 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:49:22.035 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:49:22.035 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:49:22.035 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:0 2024-09-12T06:49:22.035 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:49:22.035 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:49:22.035 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:49:22.035 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:49:22.035 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T06:49:22.036 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:49:22.036 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:49:22.036 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=0 2024-09-12T06:49:22.036 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=0 2024-09-12T06:49:22.036 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:49:22.054 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:22.054 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:22.054 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:22.054 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_32 2024-09-12T06:49:22.151 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2024-09-12T06:49:22.151 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T06:49:22.151 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:admin_socket: invalid command 2024-09-12T06:49:22.155 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:49:22.155 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_32 2024-09-12T06:49:22.155 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:1 2024-09-12T06:49:22.155 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:49:22.155 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:49:22.155 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T06:49:22.155 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:1 2024-09-12T06:49:22.155 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:49:22.155 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:49:22.155 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:49:22.155 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:49:22.155 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T06:49:22.156 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:49:22.156 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:49:22.156 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=1 2024-09-12T06:49:22.156 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:49:22.156 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T06:49:22.156 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:1 2024-09-12T06:49:22.156 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:49:22.156 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:49:22.156 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T06:49:22.156 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:1 2024-09-12T06:49:22.156 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:49:22.156 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:49:22.156 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:49:22.157 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:49:22.157 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T06:49:22.158 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:49:22.158 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:49:22.158 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=1 2024-09-12T06:49:22.158 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=1 2024-09-12T06:49:22.158 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T06:49:22.174 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:49:22.174 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:49:22.174 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:49:22.174 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_32 2024-09-12T06:49:22.273 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ return 0 2024-09-12T06:49:22.273 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T06:49:22.274 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo ' "state": "Replaying"' 2024-09-12T06:49:22.274 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep -i Replaying 2024-09-12T06:49:22.274 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: "state": "Replaying" 2024-09-12T06:49:22.276 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_state=started 2024-09-12T06:49:22.276 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test started = started 2024-09-12T06:49:22.276 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:49:22.276 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_32 2024-09-12T06:49:22.276 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:49:22.276 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:49:22.276 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:49:22.276 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:49:22.277 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_32 2024-09-12T06:49:22.277 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' journal = journal ']' 2024-09-12T06:49:22.277 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror image_32 2024-09-12T06:49:22.277 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:49:22.277 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:49:22.277 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:49:22.277 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:49:22.277 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_32 2024-09-12T06:49:22.277 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-12T06:49:22.277 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-12T06:49:22.277 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:49:22.277 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:49:22.277 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:49:22.477 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror image_32 2024-09-12T06:49:22.478 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:49:22.478 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:22.478 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_32 2024-09-12T06:49:22.478 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:49:22.478 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n image_32 ']' 2024-09-12T06:49:22.478 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/image_32' 2024-09-12T06:49:22.478 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_32 2024-09-12T06:49:22.478 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:49:22.478 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:49:22.478 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:49:22.479 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:49:22.479 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:49:22.479 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:49:22.479 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:49:22.479 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:49:22.479 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:49:22.481 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:49:22.481 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_32 2024-09-12T06:49:22.481 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:49:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:49:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:49:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:49:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:49:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:49:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:49:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:49:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:49:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:49:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:49:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:49:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:49:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:49:22.483 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:49:22.483 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:49:22.483 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:49:22.483 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:49:22.483 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:49:22.483 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:49:22.483 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:49:22.483 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:49:22.483 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:49:22.483 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:49:22.483 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:49:22.483 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:49:22.483 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:49:22.483 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:49:22.484 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:49:22.484 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:49:22.499 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:22.500 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:22.500 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:22.500 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_32 2024-09-12T06:49:22.586 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2024-09-12T06:49:22.586 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd mirror flush : flush rbd mirror 2024-09-12T06:49:22.586 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:admin_socket: invalid command 2024-09-12T06:49:22.589 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:49:22.589 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_32 2024-09-12T06:49:22.589 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:1 2024-09-12T06:49:22.589 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:49:22.589 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:49:22.589 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-12T06:49:22.589 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:1 2024-09-12T06:49:22.590 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:49:22.590 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:49:22.590 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:49:22.590 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=1 2024-09-12T06:49:22.590 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-12T06:49:22.590 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:49:22.590 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:49:22.590 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=1 2024-09-12T06:49:22.590 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:49:22.590 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-12T06:49:22.590 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:1 2024-09-12T06:49:22.590 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:49:22.590 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:49:22.590 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T06:49:22.590 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:1 2024-09-12T06:49:22.591 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:49:22.591 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:49:22.591 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:49:22.591 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:49:22.591 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T06:49:22.591 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:49:22.591 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:49:22.591 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=1 2024-09-12T06:49:22.591 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:49:22.591 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T06:49:22.607 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:49:22.608 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:49:22.608 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:49:22.608 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_32 2024-09-12T06:49:22.702 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:49:22.702 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror image_32 2024-09-12T06:49:22.702 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:49:22.702 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:49:22.702 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_32 2024-09-12T06:49:22.702 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_32 '' 2024-09-12T06:49:22.702 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:49:22.702 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:49:22.702 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_32 2024-09-12T06:49:22.702 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:49:22.702 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_32.status 2024-09-12T06:49:22.702 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_32.status 2024-09-12T06:49:22.703 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:49:22.704 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_32.status 2024-09-12T06:49:22.704 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_32 2024-09-12T06:49:22.704 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_32.status 2024-09-12T06:49:22.748 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 1 2024-09-12T06:49:22.748 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 2 2024-09-12T06:49:22.748 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:49:22.749 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6807], [object_number=6, tag_tid=1, entry_tid=6806]]], state=connected] 2024-09-12T06:49:22.749 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=7, tag_tid=1, entry_tid=6807], [object_number=6, tag_tid=1, entry_tid=6806], [object_number=5, tag_tid=1, entry_tid=6805], [object_number=4, tag_tid=1, entry_tid=6804]]], state=connected] 2024-09-12T06:49:22.751 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_32.status 2024-09-12T06:49:22.752 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-09-12T06:49:22.753 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror image_32 2024-09-12T06:49:22.754 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:49:22.754 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:49:22.754 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_32 2024-09-12T06:49:22.754 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_32 '..*' 2024-09-12T06:49:22.754 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:49:22.754 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:49:22.754 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_32 2024-09-12T06:49:22.754 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:49:22.754 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_32.status 2024-09-12T06:49:22.754 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_32.status 2024-09-12T06:49:22.754 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:49:22.755 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_32.status 2024-09-12T06:49:22.755 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_32 2024-09-12T06:49:22.755 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_32.status 2024-09-12T06:49:22.800 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 1 2024-09-12T06:49:22.800 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 2 2024-09-12T06:49:22.800 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:49:22.800 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6807], [object_number=6, tag_tid=1, entry_tid=6806]]], state=connected] 2024-09-12T06:49:22.800 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=7, tag_tid=1, entry_tid=6807], [object_number=6, tag_tid=1, entry_tid=6806], [object_number=5, tag_tid=1, entry_tid=6805], [object_number=4, tag_tid=1, entry_tid=6804]]], state=connected] 2024-09-12T06:49:22.802 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_32.status 2024-09-12T06:49:22.805 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=7, tag_tid=1, entry_tid=6807' 2024-09-12T06:49:22.805 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=7, tag_tid=1, entry_tid=6807' 2024-09-12T06:49:22.805 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 'object_number=7, tag_tid=1, entry_tid=6807' '!=' '' 2024-09-12T06:49:22.805 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ break 2024-09-12T06:49:22.805 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 'object_number=7, tag_tid=1, entry_tid=6807' = '' 2024-09-12T06:49:22.805 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ last_mirror_pos='object_number=7, tag_tid=1, entry_tid=6807' 2024-09-12T06:49:22.805 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-09-12T06:49:22.805 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-12T06:49:22.805 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:49:22.806 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_tag=2 2024-09-12T06:49:22.806 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=7, tag_tid=1, entry_tid=6807' 2024-09-12T06:49:22.806 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-12T06:49:22.806 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:49:22.807 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_tag=1 2024-09-12T06:49:22.808 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-09-12T06:49:22.808 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-12T06:49:22.808 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:49:22.809 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_entry=1 2024-09-12T06:49:22.809 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=7, tag_tid=1, entry_tid=6807' 2024-09-12T06:49:22.810 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-12T06:49:22.810 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:49:22.811 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_entry=6807 2024-09-12T06:49:22.811 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 2 = 1 -a 1 -le 6807 2024-09-12T06:49:22.811 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:49:22.811 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:49:22.811 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:49:23.013 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror image_32 2024-09-12T06:49:23.013 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:49:23.013 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:23.013 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_32 2024-09-12T06:49:23.013 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:49:23.013 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n image_32 ']' 2024-09-12T06:49:23.013 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/image_32' 2024-09-12T06:49:23.013 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_32 2024-09-12T06:49:23.013 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:49:23.014 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:49:23.014 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:49:23.014 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:49:23.014 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:49:23.014 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:49:23.014 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:49:23.014 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:49:23.014 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:49:23.017 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:49:23.017 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_32 2024-09-12T06:49:23.017 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:49:23.017 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:49:23.017 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:49:23.017 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:49:23.017 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:49:23.017 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:49:23.017 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:49:23.017 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:49:23.017 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:49:23.018 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:49:23.018 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:49:23.018 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:49:23.018 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:49:23.018 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:49:23.018 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:49:23.018 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:49:23.018 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:49:23.018 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:49:23.018 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:49:23.018 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:49:23.018 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:49:23.018 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:49:23.018 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:49:23.019 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:49:23.019 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:49:23.019 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:49:23.019 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:49:23.019 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:49:23.019 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:49:23.019 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:49:23.034 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:23.034 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:23.035 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:23.035 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_32 2024-09-12T06:49:23.122 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2024-09-12T06:49:23.122 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd mirror flush : flush rbd mirror 2024-09-12T06:49:23.122 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:admin_socket: invalid command 2024-09-12T06:49:23.125 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:49:23.125 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_32 2024-09-12T06:49:23.125 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:1 2024-09-12T06:49:23.125 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:49:23.125 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:49:23.125 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-12T06:49:23.126 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:1 2024-09-12T06:49:23.126 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:49:23.126 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:49:23.126 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:49:23.126 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=1 2024-09-12T06:49:23.126 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-12T06:49:23.126 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:49:23.126 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:49:23.126 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=1 2024-09-12T06:49:23.126 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:49:23.126 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-12T06:49:23.126 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:1 2024-09-12T06:49:23.126 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:49:23.126 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:49:23.126 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T06:49:23.127 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:1 2024-09-12T06:49:23.127 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:49:23.127 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:49:23.127 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:49:23.127 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:49:23.127 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T06:49:23.127 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:49:23.127 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:49:23.127 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=1 2024-09-12T06:49:23.127 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:49:23.127 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T06:49:23.145 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:49:23.145 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:49:23.145 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:49:23.145 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_32 2024-09-12T06:49:23.243 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:49:23.244 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror image_32 2024-09-12T06:49:23.244 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:49:23.244 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:49:23.244 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_32 2024-09-12T06:49:23.244 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_32 '' 2024-09-12T06:49:23.244 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:49:23.244 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:49:23.244 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_32 2024-09-12T06:49:23.244 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:49:23.244 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_32.status 2024-09-12T06:49:23.244 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_32.status 2024-09-12T06:49:23.244 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:49:23.245 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_32.status 2024-09-12T06:49:23.245 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_32 2024-09-12T06:49:23.245 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_32.status 2024-09-12T06:49:23.291 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 1 2024-09-12T06:49:23.291 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 2 2024-09-12T06:49:23.291 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:49:23.291 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6807], [object_number=6, tag_tid=1, entry_tid=6806]]], state=connected] 2024-09-12T06:49:23.291 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6807], [object_number=6, tag_tid=1, entry_tid=6806]]], state=connected] 2024-09-12T06:49:23.294 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_32.status 2024-09-12T06:49:23.296 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-09-12T06:49:23.296 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror image_32 2024-09-12T06:49:23.296 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:49:23.297 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:49:23.297 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_32 2024-09-12T06:49:23.297 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_32 '..*' 2024-09-12T06:49:23.297 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:49:23.297 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:49:23.297 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_32 2024-09-12T06:49:23.297 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:49:23.297 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_32.status 2024-09-12T06:49:23.297 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_32.status 2024-09-12T06:49:23.297 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:49:23.298 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_32.status 2024-09-12T06:49:23.298 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_32 2024-09-12T06:49:23.298 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_32.status 2024-09-12T06:49:23.343 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 1 2024-09-12T06:49:23.343 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 2 2024-09-12T06:49:23.343 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:49:23.343 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6807], [object_number=6, tag_tid=1, entry_tid=6806]]], state=connected] 2024-09-12T06:49:23.343 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6807], [object_number=6, tag_tid=1, entry_tid=6806]]], state=connected] 2024-09-12T06:49:23.346 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_32.status 2024-09-12T06:49:23.349 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-09-12T06:49:23.349 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-09-12T06:49:23.349 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:49:23.349 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_snap_present cluster1 mirror image_32 snap 2024-09-12T06:49:23.349 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:49:23.349 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:23.349 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_32 2024-09-12T06:49:23.349 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:49:23.349 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:49:23.349 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T06:49:23.350 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:49:24.349 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 info mirror/image_32@snap 2024-09-12T06:49:24.393 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:rbd image 'image_32': 2024-09-12T06:49:24.393 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: size 128 MiB in 32 objects 2024-09-12T06:49:24.393 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: order 22 (4 MiB objects) 2024-09-12T06:49:24.393 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: snapshot_count: 1 2024-09-12T06:49:24.394 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: id: 10af6e8dd9b7 2024-09-12T06:49:24.394 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: block_name_prefix: rbd_data.10af6e8dd9b7 2024-09-12T06:49:24.394 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: format: 2 2024-09-12T06:49:24.394 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: features: layering, exclusive-lock, journaling 2024-09-12T06:49:24.394 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: op_features: 2024-09-12T06:49:24.394 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: flags: 2024-09-12T06:49:24.395 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: create_timestamp: Thu Sep 12 06:42:58 2024 2024-09-12T06:49:24.395 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: access_timestamp: Thu Sep 12 06:42:58 2024 2024-09-12T06:49:24.395 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: modify_timestamp: Thu Sep 12 06:43:59 2024 2024-09-12T06:49:24.395 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: protected: False 2024-09-12T06:49:24.395 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: journal: 10af6e8dd9b7 2024-09-12T06:49:24.395 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring state: enabled 2024-09-12T06:49:24.395 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring mode: journal 2024-09-12T06:49:24.395 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring global id: 07f98d86-e8da-4030-a0d5-3ad5d2cf642e 2024-09-12T06:49:24.396 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring primary: false 2024-09-12T06:49:24.397 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:49:24.397 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror image_32 snap 2024-09-12T06:49:24.397 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:24.397 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_32 2024-09-12T06:49:24.397 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:49:24.398 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:49:24.398 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_32.export 2024-09-12T06:49:24.398 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_32.export 2024-09-12T06:49:24.398 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_32.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_32.export 2024-09-12T06:49:24.398 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export image_32@snap /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_32.export 2024-09-12T06:49:24.548 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:49:24.554 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export image_32@snap /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_32.export 2024-09-12T06:49:24.800 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:49:24.809 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_32.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_32.export 2024-09-12T06:49:24.869 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_32.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_32.export 2024-09-12T06:49:24.903 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:49:24.903 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:49:24.903 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_33 2024-09-12T06:49:24.903 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_snap cluster2 mirror image_33 snap 2024-09-12T06:49:24.903 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:49:24.903 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:24.903 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_33 2024-09-12T06:49:24.904 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:49:24.904 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror snap create image_33@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T06:49:26.035 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_started cluster1 mirror image_33 2024-09-12T06:49:26.035 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:49:26.035 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:26.035 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_33 2024-09-12T06:49:26.036 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_state cluster1 mirror image_33 started 2024-09-12T06:49:26.036 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:49:26.036 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:26.036 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_33 2024-09-12T06:49:26.036 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=started 2024-09-12T06:49:26.036 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:49:26.036 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:49:26.036 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:49:27.036 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_replay_state cluster1 mirror image_33 started 2024-09-12T06:49:27.036 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:49:27.036 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:27.036 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_33 2024-09-12T06:49:27.036 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=started 2024-09-12T06:49:27.037 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local status_result 2024-09-12T06:49:27.037 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=stopped 2024-09-12T06:49:27.038 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_33 2024-09-12T06:49:27.038 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_instance=cluster1 2024-09-12T06:49:27.038 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:49:27.038 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -i state 2024-09-12T06:49:27.038 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:49:27.038 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance=cluster1 2024-09-12T06:49:27.038 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local loop_instance 2024-09-12T06:49:27.038 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:49:27.039 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sleep 0 2024-09-12T06:49:27.039 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:49:27.039 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ seq 0 1 2024-09-12T06:49:27.042 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:49:27.042 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_33 2024-09-12T06:49:27.042 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:0 2024-09-12T06:49:27.043 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:49:27.043 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:49:27.043 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:49:27.043 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:49:27.043 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:49:27.043 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:49:27.043 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:49:27.043 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:49:27.043 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:49:27.043 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:49:27.043 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:49:27.043 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:49:27.043 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:49:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:49:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:0 2024-09-12T06:49:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:49:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:49:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:49:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:0 2024-09-12T06:49:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:49:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:49:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:49:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:49:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T06:49:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:49:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:49:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=0 2024-09-12T06:49:27.045 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=0 2024-09-12T06:49:27.045 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:49:27.061 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:27.061 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:27.061 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:27.061 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_33 2024-09-12T06:49:27.169 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ return 0 2024-09-12T06:49:27.169 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T06:49:27.169 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo ' "state": "Replaying"' 2024-09-12T06:49:27.169 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep -i Replaying 2024-09-12T06:49:27.170 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: "state": "Replaying" 2024-09-12T06:49:27.172 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_state=started 2024-09-12T06:49:27.172 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test started = started 2024-09-12T06:49:27.172 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:49:27.172 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_33 2024-09-12T06:49:27.172 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:49:27.172 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:49:27.172 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:49:27.172 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:49:27.172 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_33 2024-09-12T06:49:27.172 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' journal = journal ']' 2024-09-12T06:49:27.173 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror image_33 2024-09-12T06:49:27.173 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:49:27.173 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:49:27.173 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:49:27.173 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:49:27.173 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_33 2024-09-12T06:49:27.173 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-12T06:49:27.173 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-12T06:49:27.173 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:49:27.173 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:49:27.173 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:49:27.373 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror image_33 2024-09-12T06:49:27.373 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:49:27.373 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:27.373 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_33 2024-09-12T06:49:27.373 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:49:27.373 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n image_33 ']' 2024-09-12T06:49:27.374 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/image_33' 2024-09-12T06:49:27.374 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_33 2024-09-12T06:49:27.374 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:49:27.374 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:49:27.374 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:49:27.374 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:49:27.374 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:49:27.374 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:49:27.374 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:49:27.374 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:49:27.374 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:49:27.377 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:49:27.377 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_33 2024-09-12T06:49:27.377 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:49:27.377 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:49:27.377 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:49:27.377 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:49:27.378 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:49:27.378 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:49:27.378 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:49:27.378 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:49:27.378 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:49:27.378 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:49:27.378 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:49:27.378 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:49:27.378 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:49:27.378 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:49:27.378 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:49:27.378 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:49:27.378 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:49:27.378 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:49:27.379 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:49:27.379 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:49:27.379 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:49:27.379 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:49:27.379 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:49:27.379 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:49:27.379 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:49:27.379 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:49:27.379 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:49:27.379 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:49:27.379 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:49:27.379 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:49:27.394 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:27.394 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:27.394 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:27.394 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_33 2024-09-12T06:49:27.497 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:49:27.497 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror image_33 2024-09-12T06:49:27.497 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:49:27.497 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:49:27.497 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_33 2024-09-12T06:49:27.497 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_33 '' 2024-09-12T06:49:27.497 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:49:27.498 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:49:27.498 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_33 2024-09-12T06:49:27.498 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:49:27.498 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_33.status 2024-09-12T06:49:27.498 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_33.status 2024-09-12T06:49:27.498 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:49:27.498 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_33.status 2024-09-12T06:49:27.499 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_33 2024-09-12T06:49:27.499 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_33.status 2024-09-12T06:49:27.543 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 1 2024-09-12T06:49:27.543 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 1 2024-09-12T06:49:27.543 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:49:27.543 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5855], [object_number=6, tag_tid=1, entry_tid=5854]]], state=connected] 2024-09-12T06:49:27.543 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=7, tag_tid=1, entry_tid=5855], [object_number=6, tag_tid=1, entry_tid=5854], [object_number=5, tag_tid=1, entry_tid=5853], [object_number=4, tag_tid=1, entry_tid=5852]]], state=connected] 2024-09-12T06:49:27.546 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_33.status 2024-09-12T06:49:27.547 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-12T06:49:27.548 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror image_33 2024-09-12T06:49:27.549 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:49:27.549 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:49:27.549 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_33 2024-09-12T06:49:27.549 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_33 '..*' 2024-09-12T06:49:27.549 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:49:27.549 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:49:27.549 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_33 2024-09-12T06:49:27.549 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:49:27.549 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_33.status 2024-09-12T06:49:27.549 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_33.status 2024-09-12T06:49:27.549 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:49:27.550 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_33.status 2024-09-12T06:49:27.550 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_33 2024-09-12T06:49:27.550 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_33.status 2024-09-12T06:49:27.595 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 1 2024-09-12T06:49:27.595 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 1 2024-09-12T06:49:27.595 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:49:27.595 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5855], [object_number=6, tag_tid=1, entry_tid=5854]]], state=connected] 2024-09-12T06:49:27.595 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=7, tag_tid=1, entry_tid=5855], [object_number=6, tag_tid=1, entry_tid=5854], [object_number=5, tag_tid=1, entry_tid=5853], [object_number=4, tag_tid=1, entry_tid=5852]]], state=connected] 2024-09-12T06:49:27.598 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_33.status 2024-09-12T06:49:27.600 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=7, tag_tid=1, entry_tid=5855' 2024-09-12T06:49:27.600 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=7, tag_tid=1, entry_tid=5855' 2024-09-12T06:49:27.600 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 'object_number=7, tag_tid=1, entry_tid=5855' '!=' '' 2024-09-12T06:49:27.600 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ break 2024-09-12T06:49:27.601 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 'object_number=7, tag_tid=1, entry_tid=5855' = '' 2024-09-12T06:49:27.601 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ last_mirror_pos='object_number=7, tag_tid=1, entry_tid=5855' 2024-09-12T06:49:27.601 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=5, tag_tid=3, entry_tid=1' 2024-09-12T06:49:27.601 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-12T06:49:27.601 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:49:27.602 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_tag=3 2024-09-12T06:49:27.602 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=7, tag_tid=1, entry_tid=5855' 2024-09-12T06:49:27.602 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-12T06:49:27.602 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:49:27.603 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_tag=1 2024-09-12T06:49:27.603 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=5, tag_tid=3, entry_tid=1' 2024-09-12T06:49:27.603 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-12T06:49:27.603 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:49:27.604 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_entry=1 2024-09-12T06:49:27.605 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=7, tag_tid=1, entry_tid=5855' 2024-09-12T06:49:27.605 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-12T06:49:27.605 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:49:27.606 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_entry=5855 2024-09-12T06:49:27.607 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 3 = 1 -a 1 -le 5855 2024-09-12T06:49:27.607 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:49:27.607 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:49:27.607 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:49:27.808 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror image_33 2024-09-12T06:49:27.808 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:49:27.809 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:27.809 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_33 2024-09-12T06:49:27.809 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:49:27.809 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n image_33 ']' 2024-09-12T06:49:27.809 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/image_33' 2024-09-12T06:49:27.809 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_33 2024-09-12T06:49:27.809 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:49:27.809 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:49:27.809 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:49:27.809 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:49:27.809 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:49:27.809 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:49:27.809 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:49:27.809 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:49:27.810 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:49:27.813 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:49:27.813 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_33 2024-09-12T06:49:27.813 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:49:27.813 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:49:27.813 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:49:27.813 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:49:27.813 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:49:27.813 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:49:27.813 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:49:27.813 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:49:27.813 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:49:27.814 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:49:27.814 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:49:27.814 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:49:27.814 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:49:27.814 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:49:27.814 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:49:27.814 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:49:27.814 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:49:27.814 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:49:27.814 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:49:27.814 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:49:27.814 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:49:27.814 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:49:27.814 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:49:27.815 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:49:27.815 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:49:27.815 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:49:27.815 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:49:27.815 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:49:27.815 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:49:27.815 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:49:27.830 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:27.830 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:27.830 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:27.830 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_33 2024-09-12T06:49:27.928 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:49:27.929 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror image_33 2024-09-12T06:49:27.929 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:49:27.929 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:49:27.929 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_33 2024-09-12T06:49:27.929 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_33 '' 2024-09-12T06:49:27.929 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:49:27.929 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:49:27.929 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_33 2024-09-12T06:49:27.929 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:49:27.929 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_33.status 2024-09-12T06:49:27.929 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_33.status 2024-09-12T06:49:27.929 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:49:27.930 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_33.status 2024-09-12T06:49:27.930 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_33 2024-09-12T06:49:27.930 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_33.status 2024-09-12T06:49:27.976 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 1 2024-09-12T06:49:27.976 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 1 2024-09-12T06:49:27.976 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:49:27.976 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5855], [object_number=6, tag_tid=1, entry_tid=5854]]], state=connected] 2024-09-12T06:49:27.976 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5855], [object_number=6, tag_tid=1, entry_tid=5854]]], state=connected] 2024-09-12T06:49:27.978 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_33.status 2024-09-12T06:49:27.980 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-12T06:49:27.981 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror image_33 2024-09-12T06:49:27.982 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:49:27.982 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:49:27.983 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_33 2024-09-12T06:49:27.983 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_33 '..*' 2024-09-12T06:49:27.983 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:49:27.983 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:49:27.983 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_33 2024-09-12T06:49:27.983 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:49:27.983 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_33.status 2024-09-12T06:49:27.983 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_33.status 2024-09-12T06:49:27.983 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:49:27.984 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_33.status 2024-09-12T06:49:27.984 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_33 2024-09-12T06:49:27.984 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_33.status 2024-09-12T06:49:28.030 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 1 2024-09-12T06:49:28.030 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 1 2024-09-12T06:49:28.030 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:49:28.030 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5855], [object_number=6, tag_tid=1, entry_tid=5854]]], state=connected] 2024-09-12T06:49:28.030 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5855], [object_number=6, tag_tid=1, entry_tid=5854]]], state=connected] 2024-09-12T06:49:28.033 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_33.status 2024-09-12T06:49:28.035 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-12T06:49:28.035 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-09-12T06:49:28.035 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:49:28.035 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_snap_present cluster1 mirror image_33 snap 2024-09-12T06:49:28.035 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:49:28.036 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:28.036 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_33 2024-09-12T06:49:28.036 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:49:28.036 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:49:28.036 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T06:49:28.036 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:49:29.036 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 info mirror/image_33@snap 2024-09-12T06:49:29.082 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:rbd image 'image_33': 2024-09-12T06:49:29.082 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: size 128 MiB in 32 objects 2024-09-12T06:49:29.083 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: order 22 (4 MiB objects) 2024-09-12T06:49:29.083 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: snapshot_count: 1 2024-09-12T06:49:29.083 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: id: 10a8aea5a07 2024-09-12T06:49:29.083 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: block_name_prefix: rbd_data.10a8aea5a07 2024-09-12T06:49:29.083 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: format: 2 2024-09-12T06:49:29.083 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: features: layering, exclusive-lock, journaling 2024-09-12T06:49:29.083 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: op_features: 2024-09-12T06:49:29.083 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: flags: 2024-09-12T06:49:29.083 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: create_timestamp: Thu Sep 12 06:43:07 2024 2024-09-12T06:49:29.083 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: access_timestamp: Thu Sep 12 06:43:07 2024 2024-09-12T06:49:29.083 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: modify_timestamp: Thu Sep 12 06:44:08 2024 2024-09-12T06:49:29.083 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: protected: False 2024-09-12T06:49:29.083 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: journal: 10a8aea5a07 2024-09-12T06:49:29.084 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring state: enabled 2024-09-12T06:49:29.084 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring mode: journal 2024-09-12T06:49:29.084 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring global id: 8bc713dd-44f6-4b49-828c-a155600843b6 2024-09-12T06:49:29.084 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring primary: false 2024-09-12T06:49:29.086 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:49:29.086 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror image_33 snap 2024-09-12T06:49:29.086 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:29.087 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_33 2024-09-12T06:49:29.087 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:49:29.087 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:49:29.087 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_33.export 2024-09-12T06:49:29.087 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_33.export 2024-09-12T06:49:29.087 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_33.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_33.export 2024-09-12T06:49:29.088 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export image_33@snap /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_33.export 2024-09-12T06:49:29.254 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:49:29.261 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export image_33@snap /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_33.export 2024-09-12T06:49:29.464 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:49:29.472 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_33.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_33.export 2024-09-12T06:49:29.527 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_33.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_33.export 2024-09-12T06:49:29.560 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:49:29.560 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:49:29.560 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_34 2024-09-12T06:49:29.560 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_snap cluster2 mirror image_34 snap 2024-09-12T06:49:29.560 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:49:29.560 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:29.560 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_34 2024-09-12T06:49:29.561 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:49:29.561 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror snap create image_34@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T06:49:29.860 INFO:tasks.ceph.cluster2.osd.0.smithi152.stderr:2024-09-12T06:49:29.860+0000 7fd61689f640 -1 reset not still connected to 0x55e76b50edd0 2024-09-12T06:49:29.860 INFO:tasks.ceph.cluster2.osd.0.smithi152.stderr:2024-09-12T06:49:29.860+0000 7fd61689f640 -1 reset not still connected to 0x55e7772cba00 2024-09-12T06:49:29.860 INFO:tasks.ceph.cluster2.osd.0.smithi152.stderr:2024-09-12T06:49:29.861+0000 7fd61689f640 -1 reset not still connected to 0x55e77b1fcc30 2024-09-12T06:49:29.860 INFO:tasks.ceph.cluster2.osd.0.smithi152.stderr:2024-09-12T06:49:29.861+0000 7fd61689f640 -1 reset not still connected to 0x55e77b1fcd00 2024-09-12T06:49:29.866 INFO:tasks.ceph.cluster2.osd.0.smithi152.stderr:2024-09-12T06:49:29.867+0000 7fd61689f640 -1 reset not still connected to 0x55e77c768680 2024-09-12T06:49:29.866 INFO:tasks.ceph.cluster2.osd.0.smithi152.stderr:2024-09-12T06:49:29.867+0000 7fd61689f640 -1 reset not still connected to 0x55e77d6f2d00 2024-09-12T06:49:29.866 INFO:tasks.ceph.cluster2.osd.0.smithi152.stderr:2024-09-12T06:49:29.867+0000 7fd61689f640 -1 reset not still connected to 0x55e77d6f3a00 2024-09-12T06:49:29.866 INFO:tasks.ceph.cluster2.osd.0.smithi152.stderr:2024-09-12T06:49:29.867+0000 7fd61689f640 -1 reset not still connected to 0x55e782f70750 2024-09-12T06:49:29.866 INFO:tasks.ceph.cluster2.osd.0.smithi152.stderr:2024-09-12T06:49:29.867+0000 7fd61689f640 -1 reset not still connected to 0x55e7852149c0 2024-09-12T06:49:29.866 INFO:tasks.ceph.cluster2.osd.0.smithi152.stderr:2024-09-12T06:49:29.867+0000 7fd61689f640 -1 reset not still connected to 0x55e7852155f0 2024-09-12T06:49:29.867 INFO:tasks.ceph.cluster2.osd.0.smithi152.stderr:2024-09-12T06:49:29.867+0000 7fd61689f640 -1 reset not still connected to 0x55e78c9a0d00 2024-09-12T06:49:30.048 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_started cluster1 mirror image_34 2024-09-12T06:49:30.048 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:49:30.048 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:30.049 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_34 2024-09-12T06:49:30.049 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_state cluster1 mirror image_34 started 2024-09-12T06:49:30.049 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:49:30.049 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:30.049 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_34 2024-09-12T06:49:30.049 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=started 2024-09-12T06:49:30.049 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:49:30.049 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:49:30.049 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:49:31.050 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_replay_state cluster1 mirror image_34 started 2024-09-12T06:49:31.050 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:49:31.050 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:31.050 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_34 2024-09-12T06:49:31.050 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=started 2024-09-12T06:49:31.050 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local status_result 2024-09-12T06:49:31.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=stopped 2024-09-12T06:49:31.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_34 2024-09-12T06:49:31.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_instance=cluster1 2024-09-12T06:49:31.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:49:31.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -i state 2024-09-12T06:49:31.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:49:31.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance=cluster1 2024-09-12T06:49:31.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local loop_instance 2024-09-12T06:49:31.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:49:31.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sleep 0 2024-09-12T06:49:31.053 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:49:31.053 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ seq 0 1 2024-09-12T06:49:31.053 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:49:31.053 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_34 2024-09-12T06:49:31.054 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:0 2024-09-12T06:49:31.054 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:49:31.054 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:49:31.054 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:49:31.054 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:49:31.054 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:49:31.054 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:49:31.054 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:49:31.054 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:49:31.054 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:49:31.054 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:49:31.054 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:49:31.054 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:49:31.054 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:49:31.055 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:49:31.055 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:0 2024-09-12T06:49:31.055 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:49:31.055 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:49:31.056 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:49:31.056 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:0 2024-09-12T06:49:31.056 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:49:31.056 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:49:31.057 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:49:31.057 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:49:31.057 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T06:49:31.057 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:49:31.057 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:49:31.057 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=0 2024-09-12T06:49:31.057 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=0 2024-09-12T06:49:31.057 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:49:31.071 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:31.072 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:31.072 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:31.072 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_34 2024-09-12T06:49:31.167 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2024-09-12T06:49:31.167 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T06:49:31.167 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:admin_socket: invalid command 2024-09-12T06:49:31.171 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:49:31.171 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_34 2024-09-12T06:49:31.171 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:1 2024-09-12T06:49:31.171 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:49:31.171 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:49:31.171 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T06:49:31.171 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:1 2024-09-12T06:49:31.171 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:49:31.172 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:49:31.172 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:49:31.172 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:49:31.172 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T06:49:31.172 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:49:31.172 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:49:31.172 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=1 2024-09-12T06:49:31.172 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:49:31.172 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T06:49:31.172 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:1 2024-09-12T06:49:31.172 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:49:31.172 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:49:31.172 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T06:49:31.172 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:1 2024-09-12T06:49:31.173 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:49:31.173 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:49:31.173 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:49:31.173 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:49:31.173 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T06:49:31.173 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:49:31.173 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:49:31.173 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=1 2024-09-12T06:49:31.173 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=1 2024-09-12T06:49:31.173 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T06:49:31.189 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:49:31.189 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:49:31.189 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:49:31.189 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_34 2024-09-12T06:49:31.288 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ return 0 2024-09-12T06:49:31.289 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T06:49:31.289 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo ' "state": "Replaying"' 2024-09-12T06:49:31.289 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep -i Replaying 2024-09-12T06:49:31.290 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: "state": "Replaying" 2024-09-12T06:49:31.292 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_state=started 2024-09-12T06:49:31.292 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test started = started 2024-09-12T06:49:31.292 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:49:31.292 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_34 2024-09-12T06:49:31.292 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:49:31.292 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:49:31.292 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:49:31.292 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:49:31.292 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_34 2024-09-12T06:49:31.292 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' journal = journal ']' 2024-09-12T06:49:31.292 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror image_34 2024-09-12T06:49:31.292 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:49:31.292 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:49:31.293 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:49:31.293 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:49:31.293 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_34 2024-09-12T06:49:31.293 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-12T06:49:31.293 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-12T06:49:31.293 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:49:31.293 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:49:31.293 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:49:31.492 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror image_34 2024-09-12T06:49:31.493 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:49:31.493 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:31.493 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_34 2024-09-12T06:49:31.493 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:49:31.493 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n image_34 ']' 2024-09-12T06:49:31.493 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/image_34' 2024-09-12T06:49:31.493 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_34 2024-09-12T06:49:31.493 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:49:31.493 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:49:31.493 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:49:31.493 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:49:31.493 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:49:31.494 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:49:31.494 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:49:31.494 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:49:31.494 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:49:31.497 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:49:31.497 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_34 2024-09-12T06:49:31.497 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:49:31.497 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:49:31.497 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:49:31.497 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:49:31.497 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:49:31.497 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:49:31.497 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:49:31.497 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:49:31.498 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:49:31.498 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:49:31.498 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:49:31.498 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:49:31.498 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:49:31.498 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:49:31.498 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:49:31.498 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:49:31.498 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:49:31.498 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:49:31.498 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:49:31.498 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:49:31.498 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:49:31.498 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:49:31.499 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:49:31.499 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:49:31.499 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:49:31.499 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:49:31.499 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:49:31.499 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:49:31.499 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:49:31.499 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:49:31.514 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:31.514 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:31.514 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:31.514 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_34 2024-09-12T06:49:31.599 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2024-09-12T06:49:31.600 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd mirror flush : flush rbd mirror 2024-09-12T06:49:31.600 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:admin_socket: invalid command 2024-09-12T06:49:31.603 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:49:31.603 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_34 2024-09-12T06:49:31.603 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:1 2024-09-12T06:49:31.603 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:49:31.603 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:49:31.603 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-12T06:49:31.603 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:1 2024-09-12T06:49:31.603 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:49:31.603 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:49:31.603 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:49:31.604 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=1 2024-09-12T06:49:31.604 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-12T06:49:31.604 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:49:31.604 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:49:31.604 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=1 2024-09-12T06:49:31.604 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:49:31.604 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-12T06:49:31.604 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:1 2024-09-12T06:49:31.604 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:49:31.604 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:49:31.604 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T06:49:31.604 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:1 2024-09-12T06:49:31.604 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:49:31.605 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:49:31.605 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:49:31.605 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:49:31.605 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T06:49:31.605 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:49:31.605 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:49:31.605 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=1 2024-09-12T06:49:31.605 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:49:31.605 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T06:49:31.621 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:49:31.622 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:49:31.622 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:49:31.622 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_34 2024-09-12T06:49:31.717 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:49:31.717 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror image_34 2024-09-12T06:49:31.717 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:49:31.717 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:49:31.717 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_34 2024-09-12T06:49:31.717 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_34 '' 2024-09-12T06:49:31.718 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:49:31.718 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:49:31.718 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_34 2024-09-12T06:49:31.718 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:49:31.718 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_34.status 2024-09-12T06:49:31.718 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_34.status 2024-09-12T06:49:31.718 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:49:31.719 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_34.status 2024-09-12T06:49:31.719 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_34 2024-09-12T06:49:31.720 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_34.status 2024-09-12T06:49:31.763 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 1 2024-09-12T06:49:31.763 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 1 2024-09-12T06:49:31.763 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:49:31.763 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5823], [object_number=6, tag_tid=1, entry_tid=5822]]], state=connected] 2024-09-12T06:49:31.763 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5823], [object_number=6, tag_tid=1, entry_tid=5822]]], state=connected] 2024-09-12T06:49:31.765 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_34.status 2024-09-12T06:49:31.767 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-12T06:49:31.768 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror image_34 2024-09-12T06:49:31.768 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:49:31.768 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:49:31.768 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_34 2024-09-12T06:49:31.769 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_34 '..*' 2024-09-12T06:49:31.769 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:49:31.769 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:49:31.769 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_34 2024-09-12T06:49:31.769 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:49:31.769 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_34.status 2024-09-12T06:49:31.769 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_34.status 2024-09-12T06:49:31.769 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:49:31.770 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_34.status 2024-09-12T06:49:31.770 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_34 2024-09-12T06:49:31.770 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_34.status 2024-09-12T06:49:31.815 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 1 2024-09-12T06:49:31.815 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 1 2024-09-12T06:49:31.815 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:49:31.815 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5823], [object_number=6, tag_tid=1, entry_tid=5822]]], state=connected] 2024-09-12T06:49:31.815 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5823], [object_number=6, tag_tid=1, entry_tid=5822]]], state=connected] 2024-09-12T06:49:31.818 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_34.status 2024-09-12T06:49:31.822 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-09-12T06:49:31.822 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-09-12T06:49:31.822 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 'object_number=5, tag_tid=2, entry_tid=1' '!=' '' 2024-09-12T06:49:31.822 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ break 2024-09-12T06:49:31.822 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 'object_number=5, tag_tid=2, entry_tid=1' = '' 2024-09-12T06:49:31.822 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ last_mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-09-12T06:49:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=5, tag_tid=3, entry_tid=1' 2024-09-12T06:49:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-12T06:49:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:49:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_tag=3 2024-09-12T06:49:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-09-12T06:49:31.824 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-12T06:49:31.824 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:49:31.826 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_tag=2 2024-09-12T06:49:31.827 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=5, tag_tid=3, entry_tid=1' 2024-09-12T06:49:31.827 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-12T06:49:31.827 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:49:31.827 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_entry=1 2024-09-12T06:49:31.827 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-09-12T06:49:31.827 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-12T06:49:31.828 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:49:31.828 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_entry=1 2024-09-12T06:49:31.828 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 3 = 2 -a 1 -le 1 2024-09-12T06:49:31.828 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:49:31.828 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:49:31.828 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:49:32.031 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror image_34 2024-09-12T06:49:32.031 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:49:32.031 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:32.031 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_34 2024-09-12T06:49:32.031 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:49:32.032 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n image_34 ']' 2024-09-12T06:49:32.032 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/image_34' 2024-09-12T06:49:32.032 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_34 2024-09-12T06:49:32.032 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:49:32.032 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:49:32.032 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:49:32.032 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:49:32.032 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:49:32.032 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:49:32.032 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:49:32.032 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:49:32.032 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:49:32.033 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:49:32.033 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_34 2024-09-12T06:49:32.033 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:49:32.033 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:49:32.033 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:49:32.033 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:49:32.033 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:49:32.033 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:49:32.033 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:49:32.033 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:49:32.033 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:49:32.033 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:49:32.033 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:49:32.033 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:49:32.034 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:49:32.034 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:49:32.034 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:49:32.034 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:49:32.034 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:49:32.034 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:49:32.034 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:49:32.034 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:49:32.034 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:49:32.034 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:49:32.034 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:49:32.034 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:49:32.035 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:49:32.035 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:49:32.035 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:49:32.035 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:49:32.035 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:49:32.035 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:49:32.049 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:32.049 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:32.049 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:32.049 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_34 2024-09-12T06:49:32.137 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2024-09-12T06:49:32.137 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd mirror flush : flush rbd mirror 2024-09-12T06:49:32.137 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:admin_socket: invalid command 2024-09-12T06:49:32.143 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:49:32.143 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_34 2024-09-12T06:49:32.144 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:1 2024-09-12T06:49:32.144 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:49:32.144 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:49:32.144 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-12T06:49:32.144 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:1 2024-09-12T06:49:32.144 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:49:32.145 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:49:32.145 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:49:32.145 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=1 2024-09-12T06:49:32.145 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-12T06:49:32.145 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:49:32.145 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:49:32.145 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=1 2024-09-12T06:49:32.146 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:49:32.146 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-12T06:49:32.146 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:1 2024-09-12T06:49:32.146 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:49:32.146 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:49:32.146 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T06:49:32.147 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:1 2024-09-12T06:49:32.147 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:49:32.147 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:49:32.147 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:49:32.147 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:49:32.147 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T06:49:32.148 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:49:32.148 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:49:32.148 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=1 2024-09-12T06:49:32.148 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:49:32.148 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T06:49:32.160 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:49:32.160 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:49:32.160 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:49:32.160 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_34 2024-09-12T06:49:32.261 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:49:32.262 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror image_34 2024-09-12T06:49:32.263 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:49:32.263 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:49:32.263 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_34 2024-09-12T06:49:32.263 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_34 '' 2024-09-12T06:49:32.263 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:49:32.263 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:49:32.263 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_34 2024-09-12T06:49:32.263 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:49:32.263 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_34.status 2024-09-12T06:49:32.263 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_34.status 2024-09-12T06:49:32.263 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:49:32.264 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_34.status 2024-09-12T06:49:32.264 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_34 2024-09-12T06:49:32.264 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_34.status 2024-09-12T06:49:32.309 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 1 2024-09-12T06:49:32.310 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 1 2024-09-12T06:49:32.310 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:49:32.310 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5823], [object_number=6, tag_tid=1, entry_tid=5822]]], state=connected] 2024-09-12T06:49:32.310 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5823], [object_number=6, tag_tid=1, entry_tid=5822]]], state=connected] 2024-09-12T06:49:32.368 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_34.status 2024-09-12T06:49:32.369 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-12T06:49:32.370 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror image_34 2024-09-12T06:49:32.370 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:49:32.370 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:49:32.370 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_34 2024-09-12T06:49:32.371 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_34 '..*' 2024-09-12T06:49:32.371 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:49:32.371 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:49:32.371 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_34 2024-09-12T06:49:32.371 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:49:32.371 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_34.status 2024-09-12T06:49:32.371 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_34.status 2024-09-12T06:49:32.371 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:49:32.372 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_34.status 2024-09-12T06:49:32.372 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_34 2024-09-12T06:49:32.372 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_34.status 2024-09-12T06:49:32.417 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 1 2024-09-12T06:49:32.417 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 1 2024-09-12T06:49:32.417 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:49:32.417 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5823], [object_number=6, tag_tid=1, entry_tid=5822]]], state=connected] 2024-09-12T06:49:32.418 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5823], [object_number=6, tag_tid=1, entry_tid=5822]]], state=connected] 2024-09-12T06:49:32.460 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_34.status 2024-09-12T06:49:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-12T06:49:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-09-12T06:49:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:49:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_snap_present cluster1 mirror image_34 snap 2024-09-12T06:49:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:49:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:32.463 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_34 2024-09-12T06:49:32.463 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:49:32.463 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:49:32.463 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T06:49:32.463 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:49:33.463 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 info mirror/image_34@snap 2024-09-12T06:49:33.512 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:rbd image 'image_34': 2024-09-12T06:49:33.513 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: size 128 MiB in 32 objects 2024-09-12T06:49:33.513 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: order 22 (4 MiB objects) 2024-09-12T06:49:33.513 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: snapshot_count: 1 2024-09-12T06:49:33.513 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: id: 10af5d8605d2 2024-09-12T06:49:33.513 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: block_name_prefix: rbd_data.10af5d8605d2 2024-09-12T06:49:33.513 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: format: 2 2024-09-12T06:49:33.513 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: features: layering, exclusive-lock, journaling 2024-09-12T06:49:33.513 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: op_features: 2024-09-12T06:49:33.513 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: flags: 2024-09-12T06:49:33.513 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: create_timestamp: Thu Sep 12 06:43:14 2024 2024-09-12T06:49:33.513 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: access_timestamp: Thu Sep 12 06:43:14 2024 2024-09-12T06:49:33.514 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: modify_timestamp: Thu Sep 12 06:44:15 2024 2024-09-12T06:49:33.514 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: protected: False 2024-09-12T06:49:33.514 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: journal: 10af5d8605d2 2024-09-12T06:49:33.514 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring state: enabled 2024-09-12T06:49:33.514 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring mode: journal 2024-09-12T06:49:33.514 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring global id: 103ec6a3-ee3d-4cf9-949b-d21fe832424d 2024-09-12T06:49:33.514 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring primary: false 2024-09-12T06:49:33.516 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:49:33.516 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror image_34 snap 2024-09-12T06:49:33.516 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:33.517 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_34 2024-09-12T06:49:33.517 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:49:33.517 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:49:33.517 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_34.export 2024-09-12T06:49:33.517 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_34.export 2024-09-12T06:49:33.517 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_34.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_34.export 2024-09-12T06:49:33.517 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export image_34@snap /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_34.export 2024-09-12T06:49:33.671 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:49:33.679 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export image_34@snap /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_34.export 2024-09-12T06:49:33.851 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:49:33.858 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_34.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_34.export 2024-09-12T06:49:33.914 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_34.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_34.export 2024-09-12T06:49:33.946 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:49:33.946 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:49:33.947 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_35 2024-09-12T06:49:33.947 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_snap cluster2 mirror image_35 snap 2024-09-12T06:49:33.947 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:49:33.947 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:33.947 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_35 2024-09-12T06:49:33.947 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:49:33.947 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror snap create image_35@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T06:49:35.491 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_started cluster1 mirror image_35 2024-09-12T06:49:35.491 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:49:35.491 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:35.491 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_35 2024-09-12T06:49:35.491 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_state cluster1 mirror image_35 started 2024-09-12T06:49:35.491 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:49:35.491 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:35.491 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_35 2024-09-12T06:49:35.491 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=started 2024-09-12T06:49:35.492 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:49:35.492 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:49:35.492 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:49:36.493 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_replay_state cluster1 mirror image_35 started 2024-09-12T06:49:36.493 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:49:36.493 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:36.493 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_35 2024-09-12T06:49:36.493 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=started 2024-09-12T06:49:36.493 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local status_result 2024-09-12T06:49:36.493 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=stopped 2024-09-12T06:49:36.493 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_35 2024-09-12T06:49:36.493 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -i state 2024-09-12T06:49:36.494 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_instance=cluster1 2024-09-12T06:49:36.494 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:49:36.494 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:49:36.494 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance=cluster1 2024-09-12T06:49:36.494 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local loop_instance 2024-09-12T06:49:36.494 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:49:36.494 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sleep 0 2024-09-12T06:49:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:49:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ seq 0 1 2024-09-12T06:49:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:49:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_35 2024-09-12T06:49:36.498 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:0 2024-09-12T06:49:36.498 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:49:36.498 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:49:36.498 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:49:36.498 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:49:36.498 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:49:36.498 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:49:36.498 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:49:36.498 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:49:36.498 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:49:36.498 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:49:36.498 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:49:36.498 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:49:36.498 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:49:36.499 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:49:36.499 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:0 2024-09-12T06:49:36.499 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:49:36.499 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:49:36.499 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:49:36.499 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:0 2024-09-12T06:49:36.499 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:49:36.499 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:49:36.499 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:49:36.499 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:49:36.499 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T06:49:36.499 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:49:36.499 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:49:36.500 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=0 2024-09-12T06:49:36.500 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=0 2024-09-12T06:49:36.500 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:49:36.520 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:36.520 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:36.520 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:36.520 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_35 2024-09-12T06:49:36.624 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ return 0 2024-09-12T06:49:36.624 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T06:49:36.624 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo ' "state": "Replaying"' 2024-09-12T06:49:36.625 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep -i Replaying 2024-09-12T06:49:36.625 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: "state": "Replaying" 2024-09-12T06:49:36.627 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_state=started 2024-09-12T06:49:36.627 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test started = started 2024-09-12T06:49:36.627 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:49:36.627 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_35 2024-09-12T06:49:36.627 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:49:36.627 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:49:36.628 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:49:36.628 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:49:36.628 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_35 2024-09-12T06:49:36.628 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' journal = journal ']' 2024-09-12T06:49:36.628 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror image_35 2024-09-12T06:49:36.628 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:49:36.628 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:49:36.628 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:49:36.628 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:49:36.628 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_35 2024-09-12T06:49:36.628 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-12T06:49:36.628 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-12T06:49:36.628 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:49:36.628 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:49:36.629 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:49:36.828 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror image_35 2024-09-12T06:49:36.828 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:49:36.828 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:36.828 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_35 2024-09-12T06:49:36.828 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:49:36.829 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n image_35 ']' 2024-09-12T06:49:36.829 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/image_35' 2024-09-12T06:49:36.829 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_35 2024-09-12T06:49:36.829 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:49:36.829 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:49:36.829 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:49:36.829 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:49:36.829 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:49:36.829 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:49:36.829 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:49:36.829 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:49:36.829 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:49:36.832 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:49:36.832 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_35 2024-09-12T06:49:36.832 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:49:36.832 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:49:36.833 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:49:36.833 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:49:36.833 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:49:36.833 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:49:36.833 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:49:36.833 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:49:36.833 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:49:36.833 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:49:36.833 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:49:36.833 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:49:36.833 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:49:36.833 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:49:36.833 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:49:36.834 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:49:36.834 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:49:36.834 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:49:36.834 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:49:36.834 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:49:36.834 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:49:36.834 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:49:36.834 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:49:36.834 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:49:36.834 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:49:36.834 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:49:36.834 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:49:36.834 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:49:36.834 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:49:36.834 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:49:36.852 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:36.852 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:36.852 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:36.853 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_35 2024-09-12T06:49:36.951 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:49:36.951 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror image_35 2024-09-12T06:49:36.952 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:49:36.952 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:49:36.952 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_35 2024-09-12T06:49:36.952 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_35 '' 2024-09-12T06:49:36.952 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:49:36.952 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:49:36.952 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_35 2024-09-12T06:49:36.952 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:49:36.952 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_35.status 2024-09-12T06:49:36.952 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_35.status 2024-09-12T06:49:36.952 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:49:36.953 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_35.status 2024-09-12T06:49:36.953 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_35 2024-09-12T06:49:36.953 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_35.status 2024-09-12T06:49:36.998 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 1 2024-09-12T06:49:36.998 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 1 2024-09-12T06:49:36.998 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:49:36.998 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5875], [object_number=6, tag_tid=1, entry_tid=5874]]], state=connected] 2024-09-12T06:49:36.998 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5875], [object_number=6, tag_tid=1, entry_tid=5874]]], state=connected] 2024-09-12T06:49:37.001 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_35.status 2024-09-12T06:49:37.003 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-09-12T06:49:37.004 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror image_35 2024-09-12T06:49:37.004 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:49:37.004 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:49:37.004 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_35 2024-09-12T06:49:37.004 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_35 '..*' 2024-09-12T06:49:37.004 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:49:37.004 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:49:37.004 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_35 2024-09-12T06:49:37.005 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:49:37.005 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_35.status 2024-09-12T06:49:37.005 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_35.status 2024-09-12T06:49:37.005 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:49:37.005 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_35.status 2024-09-12T06:49:37.005 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_35 2024-09-12T06:49:37.006 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_35.status 2024-09-12T06:49:37.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 1 2024-09-12T06:49:37.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 1 2024-09-12T06:49:37.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:49:37.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5875], [object_number=6, tag_tid=1, entry_tid=5874]]], state=connected] 2024-09-12T06:49:37.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5875], [object_number=6, tag_tid=1, entry_tid=5874]]], state=connected] 2024-09-12T06:49:37.053 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_35.status 2024-09-12T06:49:37.056 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-09-12T06:49:37.056 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-09-12T06:49:37.056 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:49:37.056 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_snap_present cluster1 mirror image_35 snap 2024-09-12T06:49:37.056 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:49:37.056 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:37.056 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_35 2024-09-12T06:49:37.056 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:49:37.056 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:49:37.057 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T06:49:37.057 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:49:38.056 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 info mirror/image_35@snap 2024-09-12T06:49:38.105 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:rbd image 'image_35': 2024-09-12T06:49:38.105 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: size 128 MiB in 32 objects 2024-09-12T06:49:38.105 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: order 22 (4 MiB objects) 2024-09-12T06:49:38.105 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: snapshot_count: 1 2024-09-12T06:49:38.105 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: id: 10a8452e01d2 2024-09-12T06:49:38.105 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: block_name_prefix: rbd_data.10a8452e01d2 2024-09-12T06:49:38.105 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: format: 2 2024-09-12T06:49:38.105 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: features: layering, exclusive-lock, journaling 2024-09-12T06:49:38.105 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: op_features: 2024-09-12T06:49:38.105 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: flags: 2024-09-12T06:49:38.105 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: create_timestamp: Thu Sep 12 06:43:19 2024 2024-09-12T06:49:38.106 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: access_timestamp: Thu Sep 12 06:43:19 2024 2024-09-12T06:49:38.106 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: modify_timestamp: Thu Sep 12 06:43:19 2024 2024-09-12T06:49:38.106 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: protected: False 2024-09-12T06:49:38.106 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: journal: 10a8452e01d2 2024-09-12T06:49:38.106 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring state: enabled 2024-09-12T06:49:38.106 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring mode: journal 2024-09-12T06:49:38.106 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring global id: 99a81395-d110-4e38-b767-46a43d1331a0 2024-09-12T06:49:38.106 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring primary: false 2024-09-12T06:49:38.108 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:49:38.108 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror image_35 snap 2024-09-12T06:49:38.109 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:38.109 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_35 2024-09-12T06:49:38.109 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:49:38.109 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:49:38.109 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_35.export 2024-09-12T06:49:38.109 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_35.export 2024-09-12T06:49:38.109 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_35.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_35.export 2024-09-12T06:49:38.109 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export image_35@snap /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_35.export 2024-09-12T06:49:38.258 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:49:38.265 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export image_35@snap /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_35.export 2024-09-12T06:49:38.468 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:49:38.474 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_35.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_35.export 2024-09-12T06:49:38.531 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_35.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_35.export 2024-09-12T06:49:38.564 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:49:38.564 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:49:38.564 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_36 2024-09-12T06:49:38.564 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_snap cluster2 mirror image_36 snap 2024-09-12T06:49:38.564 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:49:38.565 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:38.565 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_36 2024-09-12T06:49:38.565 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:49:38.565 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror snap create image_36@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T06:49:39.515 INFO:tasks.ceph.cluster2.osd.0.smithi152.stderr:2024-09-12T06:49:39.516+0000 7fd606c6e640 -1 bdev(0x55e762622c00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 2 2024-09-12T06:49:40.511 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_started cluster1 mirror image_36 2024-09-12T06:49:40.511 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:49:40.511 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:40.511 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_36 2024-09-12T06:49:40.511 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_state cluster1 mirror image_36 started 2024-09-12T06:49:40.511 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:49:40.511 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:40.511 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_36 2024-09-12T06:49:40.511 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=started 2024-09-12T06:49:40.511 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:49:40.512 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:49:40.512 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:49:41.517 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_replay_state cluster1 mirror image_36 started 2024-09-12T06:49:41.517 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:49:41.517 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:41.518 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_36 2024-09-12T06:49:41.518 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=started 2024-09-12T06:49:41.518 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local status_result 2024-09-12T06:49:41.518 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=stopped 2024-09-12T06:49:41.518 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_36 2024-09-12T06:49:41.518 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_instance=cluster1 2024-09-12T06:49:41.518 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:49:41.518 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -i state 2024-09-12T06:49:41.518 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:49:41.518 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance=cluster1 2024-09-12T06:49:41.518 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local loop_instance 2024-09-12T06:49:41.519 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:49:41.519 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sleep 0 2024-09-12T06:49:41.519 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:49:41.519 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ seq 0 1 2024-09-12T06:49:41.519 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:49:41.519 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_36 2024-09-12T06:49:41.519 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:0 2024-09-12T06:49:41.519 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:49:41.519 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:49:41.519 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:49:41.520 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:49:41.520 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:49:41.520 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:49:41.520 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:49:41.520 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:49:41.520 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:49:41.520 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:49:41.520 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:49:41.520 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:49:41.520 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:49:41.521 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:49:41.521 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:0 2024-09-12T06:49:41.521 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:49:41.521 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:49:41.521 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:49:41.521 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:0 2024-09-12T06:49:41.521 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:49:41.521 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:49:41.521 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:49:41.521 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:49:41.521 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T06:49:41.522 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:49:41.522 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:49:41.522 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=0 2024-09-12T06:49:41.523 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=0 2024-09-12T06:49:41.523 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:49:41.534 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:41.534 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:41.534 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:41.535 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_36 2024-09-12T06:49:41.626 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2024-09-12T06:49:41.627 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T06:49:41.627 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:admin_socket: invalid command 2024-09-12T06:49:41.630 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:49:41.630 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_36 2024-09-12T06:49:41.630 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:1 2024-09-12T06:49:41.630 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:49:41.630 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:49:41.630 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T06:49:41.630 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:1 2024-09-12T06:49:41.631 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:49:41.631 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:49:41.631 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:49:41.631 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:49:41.631 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T06:49:41.631 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:49:41.631 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:49:41.631 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=1 2024-09-12T06:49:41.631 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:49:41.631 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T06:49:41.631 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:1 2024-09-12T06:49:41.631 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:49:41.631 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:49:41.631 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T06:49:41.632 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:1 2024-09-12T06:49:41.632 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:49:41.632 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:49:41.632 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:49:41.632 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:49:41.632 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T06:49:41.632 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:49:41.632 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:49:41.632 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=1 2024-09-12T06:49:41.632 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=1 2024-09-12T06:49:41.632 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T06:49:41.647 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:49:41.647 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:49:41.647 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:49:41.647 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_36 2024-09-12T06:49:41.746 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ return 0 2024-09-12T06:49:41.747 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T06:49:41.747 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo ' "state": "Replaying"' 2024-09-12T06:49:41.747 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep -i Replaying 2024-09-12T06:49:41.748 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: "state": "Replaying" 2024-09-12T06:49:41.750 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_state=started 2024-09-12T06:49:41.750 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test started = started 2024-09-12T06:49:41.750 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:49:41.750 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_36 2024-09-12T06:49:41.750 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:49:41.750 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:49:41.750 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:49:41.750 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:49:41.750 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_36 2024-09-12T06:49:41.750 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' journal = journal ']' 2024-09-12T06:49:41.751 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror image_36 2024-09-12T06:49:41.751 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:49:41.751 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:49:41.751 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:49:41.751 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:49:41.751 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_36 2024-09-12T06:49:41.751 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-12T06:49:41.751 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-12T06:49:41.751 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:49:41.751 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:49:41.751 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:49:41.950 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror image_36 2024-09-12T06:49:41.951 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:49:41.951 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:41.951 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_36 2024-09-12T06:49:41.951 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:49:41.951 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n image_36 ']' 2024-09-12T06:49:41.951 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/image_36' 2024-09-12T06:49:41.951 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_36 2024-09-12T06:49:41.951 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:49:41.951 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:49:41.951 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:49:41.951 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:49:41.952 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:49:41.952 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:49:41.952 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:49:41.952 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:49:41.955 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:49:41.955 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:49:41.955 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_36 2024-09-12T06:49:41.955 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:49:41.955 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:49:41.955 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:49:41.955 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:49:41.955 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:49:41.955 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:49:41.955 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:49:41.956 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:49:41.956 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:49:41.956 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:49:41.956 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:49:41.956 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:49:41.956 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:49:41.956 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:49:41.956 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:49:41.956 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:49:41.956 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:49:41.956 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:49:41.956 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:49:41.956 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:49:41.956 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:49:41.957 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:49:41.957 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:49:41.957 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:49:41.957 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:49:41.957 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:49:41.957 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:49:41.957 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:49:41.957 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:49:41.957 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:49:41.972 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:41.972 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:41.973 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:41.973 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_36 2024-09-12T06:49:42.058 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2024-09-12T06:49:42.058 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd mirror flush : flush rbd mirror 2024-09-12T06:49:42.058 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:admin_socket: invalid command 2024-09-12T06:49:42.061 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:49:42.062 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_36 2024-09-12T06:49:42.062 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:1 2024-09-12T06:49:42.062 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:49:42.062 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:49:42.062 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-12T06:49:42.062 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:1 2024-09-12T06:49:42.062 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:49:42.062 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:49:42.062 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:49:42.062 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=1 2024-09-12T06:49:42.062 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-12T06:49:42.062 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:49:42.062 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:49:42.063 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=1 2024-09-12T06:49:42.063 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:49:42.063 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-12T06:49:42.063 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:1 2024-09-12T06:49:42.063 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:49:42.063 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:49:42.063 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T06:49:42.063 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:1 2024-09-12T06:49:42.063 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:49:42.063 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:49:42.063 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:49:42.063 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:49:42.063 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T06:49:42.063 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:49:42.064 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:49:42.064 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=1 2024-09-12T06:49:42.064 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:49:42.064 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T06:49:42.081 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:49:42.081 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:49:42.081 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:49:42.081 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_36 2024-09-12T06:49:42.181 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:49:42.181 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror image_36 2024-09-12T06:49:42.182 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:49:42.182 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:49:42.182 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_36 2024-09-12T06:49:42.182 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_36 '' 2024-09-12T06:49:42.182 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:49:42.182 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:49:42.182 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_36 2024-09-12T06:49:42.182 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:49:42.182 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_36.status 2024-09-12T06:49:42.182 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_36.status 2024-09-12T06:49:42.183 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:49:42.183 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_36.status 2024-09-12T06:49:42.183 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_36 2024-09-12T06:49:42.183 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_36.status 2024-09-12T06:49:42.229 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 1 2024-09-12T06:49:42.229 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 1 2024-09-12T06:49:42.229 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:49:42.229 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=6122], [object_number=7, tag_tid=1, entry_tid=6119]]], state=connected] 2024-09-12T06:49:42.230 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=6122], [object_number=7, tag_tid=1, entry_tid=6119]]], state=connected] 2024-09-12T06:49:42.232 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_36.status 2024-09-12T06:49:42.233 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-09-12T06:49:42.234 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror image_36 2024-09-12T06:49:42.235 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:49:42.235 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:49:42.235 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_36 2024-09-12T06:49:42.235 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_36 '..*' 2024-09-12T06:49:42.235 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:49:42.235 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:49:42.235 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_36 2024-09-12T06:49:42.235 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:49:42.235 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_36.status 2024-09-12T06:49:42.235 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_36.status 2024-09-12T06:49:42.235 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:49:42.236 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_36.status 2024-09-12T06:49:42.236 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_36 2024-09-12T06:49:42.236 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_36.status 2024-09-12T06:49:42.282 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 1 2024-09-12T06:49:42.282 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 1 2024-09-12T06:49:42.282 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:49:42.282 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=6122], [object_number=7, tag_tid=1, entry_tid=6119]]], state=connected] 2024-09-12T06:49:42.282 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=6122], [object_number=7, tag_tid=1, entry_tid=6119]]], state=connected] 2024-09-12T06:49:42.285 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_36.status 2024-09-12T06:49:42.287 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-09-12T06:49:42.287 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-09-12T06:49:42.287 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:49:42.287 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_snap_present cluster1 mirror image_36 snap 2024-09-12T06:49:42.287 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:49:42.287 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:42.287 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_36 2024-09-12T06:49:42.287 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:49:42.287 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:49:42.287 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T06:49:42.288 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:49:43.288 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 info mirror/image_36@snap 2024-09-12T06:49:43.334 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:rbd image 'image_36': 2024-09-12T06:49:43.334 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: size 128 MiB in 32 objects 2024-09-12T06:49:43.334 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: order 22 (4 MiB objects) 2024-09-12T06:49:43.334 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: snapshot_count: 1 2024-09-12T06:49:43.334 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: id: 10af74ba0de7 2024-09-12T06:49:43.335 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: block_name_prefix: rbd_data.10af74ba0de7 2024-09-12T06:49:43.335 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: format: 2 2024-09-12T06:49:43.335 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: features: layering, exclusive-lock, journaling 2024-09-12T06:49:43.335 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: op_features: 2024-09-12T06:49:43.335 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: flags: 2024-09-12T06:49:43.335 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: create_timestamp: Thu Sep 12 06:43:27 2024 2024-09-12T06:49:43.335 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: access_timestamp: Thu Sep 12 06:43:27 2024 2024-09-12T06:49:43.335 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: modify_timestamp: Thu Sep 12 06:43:27 2024 2024-09-12T06:49:43.335 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: protected: False 2024-09-12T06:49:43.335 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: journal: 10af74ba0de7 2024-09-12T06:49:43.335 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring state: enabled 2024-09-12T06:49:43.335 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring mode: journal 2024-09-12T06:49:43.335 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring global id: b928c413-0846-418a-a6ee-cf9968489a50 2024-09-12T06:49:43.335 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring primary: false 2024-09-12T06:49:43.338 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:49:43.338 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror image_36 snap 2024-09-12T06:49:43.338 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:43.338 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_36 2024-09-12T06:49:43.338 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:49:43.338 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:49:43.338 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_36.export 2024-09-12T06:49:43.338 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_36.export 2024-09-12T06:49:43.338 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_36.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_36.export 2024-09-12T06:49:43.339 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export image_36@snap /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_36.export 2024-09-12T06:49:43.488 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:49:43.495 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export image_36@snap /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_36.export 2024-09-12T06:49:43.724 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:49:43.731 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_36.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_36.export 2024-09-12T06:49:43.786 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_36.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_36.export 2024-09-12T06:49:43.817 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:49:43.817 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:49:43.817 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_37 2024-09-12T06:49:43.817 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_snap cluster2 mirror image_37 snap 2024-09-12T06:49:43.817 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:49:43.817 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:43.818 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_37 2024-09-12T06:49:43.818 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:49:43.818 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror snap create image_37@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T06:49:44.907 INFO:tasks.ceph.cluster2.osd.0.smithi152.stderr:2024-09-12T06:49:44.908+0000 7fd61689f640 -1 reset not still connected to 0x55e785c50b60 2024-09-12T06:49:44.908 INFO:tasks.ceph.cluster2.osd.0.smithi152.stderr:2024-09-12T06:49:44.909+0000 7fd61689f640 -1 reset not still connected to 0x55e785c51520 2024-09-12T06:49:44.908 INFO:tasks.ceph.cluster2.osd.0.smithi152.stderr:2024-09-12T06:49:44.909+0000 7fd61689f640 -1 reset not still connected to 0x55e78805cd00 2024-09-12T06:49:45.534 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_started cluster1 mirror image_37 2024-09-12T06:49:45.534 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:49:45.534 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:45.534 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_37 2024-09-12T06:49:45.534 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_state cluster1 mirror image_37 started 2024-09-12T06:49:45.535 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:49:45.535 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:45.535 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_37 2024-09-12T06:49:45.535 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=started 2024-09-12T06:49:45.535 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:49:45.535 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:49:45.535 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:49:46.535 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_replay_state cluster1 mirror image_37 started 2024-09-12T06:49:46.535 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:49:46.535 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:46.535 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_37 2024-09-12T06:49:46.536 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=started 2024-09-12T06:49:46.536 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local status_result 2024-09-12T06:49:46.536 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=stopped 2024-09-12T06:49:46.540 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_37 2024-09-12T06:49:46.540 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -i state 2024-09-12T06:49:46.540 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_instance=cluster1 2024-09-12T06:49:46.540 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:49:46.540 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:49:46.541 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance=cluster1 2024-09-12T06:49:46.541 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local loop_instance 2024-09-12T06:49:46.541 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:49:46.541 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sleep 0 2024-09-12T06:49:46.541 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:49:46.541 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ seq 0 1 2024-09-12T06:49:46.541 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:49:46.541 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_37 2024-09-12T06:49:46.541 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:0 2024-09-12T06:49:46.541 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:49:46.541 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:49:46.541 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:49:46.542 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:49:46.542 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:49:46.542 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:49:46.542 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:49:46.542 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:49:46.542 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:49:46.542 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:49:46.542 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:49:46.542 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:49:46.542 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:49:46.542 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:49:46.542 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:0 2024-09-12T06:49:46.542 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:49:46.543 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:49:46.543 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:49:46.543 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:0 2024-09-12T06:49:46.543 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:49:46.543 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:49:46.543 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:49:46.543 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:49:46.543 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T06:49:46.543 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:49:46.543 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:49:46.543 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=0 2024-09-12T06:49:46.543 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=0 2024-09-12T06:49:46.543 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:49:46.558 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:46.558 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:46.558 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:46.558 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_37 2024-09-12T06:49:46.657 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ return 0 2024-09-12T06:49:46.658 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T06:49:46.658 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo ' "state": "Replaying"' 2024-09-12T06:49:46.658 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep -i Replaying 2024-09-12T06:49:46.658 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: "state": "Replaying" 2024-09-12T06:49:46.660 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_state=started 2024-09-12T06:49:46.660 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test started = started 2024-09-12T06:49:46.660 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:49:46.660 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_37 2024-09-12T06:49:46.661 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:49:46.661 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:49:46.661 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:49:46.661 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:49:46.661 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_37 2024-09-12T06:49:46.661 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' journal = journal ']' 2024-09-12T06:49:46.661 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror image_37 2024-09-12T06:49:46.661 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:49:46.661 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:49:46.661 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:49:46.662 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:49:46.662 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_37 2024-09-12T06:49:46.662 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-12T06:49:46.662 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-12T06:49:46.662 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:49:46.662 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:49:46.662 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:49:46.861 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror image_37 2024-09-12T06:49:46.861 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:49:46.861 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:46.862 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_37 2024-09-12T06:49:46.862 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:49:46.862 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n image_37 ']' 2024-09-12T06:49:46.862 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/image_37' 2024-09-12T06:49:46.862 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_37 2024-09-12T06:49:46.862 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:49:46.862 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:49:46.862 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:49:46.862 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:49:46.862 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:49:46.862 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:49:46.863 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:49:46.863 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:49:46.863 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:49:46.866 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:49:46.866 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_37 2024-09-12T06:49:46.866 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:49:46.866 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:49:46.866 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:49:46.866 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:49:46.867 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:49:46.867 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:49:46.867 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:49:46.867 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:49:46.867 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:49:46.867 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:49:46.867 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:49:46.867 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:49:46.867 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:49:46.867 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:49:46.867 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:49:46.868 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:49:46.868 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:49:46.868 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:49:46.868 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:49:46.868 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:49:46.868 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:49:46.868 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:49:46.868 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:49:46.868 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:49:46.868 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:49:46.868 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:49:46.868 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:49:46.869 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:49:46.869 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:49:46.869 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:49:46.883 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:46.883 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:46.883 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:46.883 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_37 2024-09-12T06:49:46.982 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:49:46.982 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror image_37 2024-09-12T06:49:46.982 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:49:46.982 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:49:46.982 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_37 2024-09-12T06:49:46.982 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_37 '' 2024-09-12T06:49:46.982 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:49:46.982 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:49:46.983 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_37 2024-09-12T06:49:46.983 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:49:46.983 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_37.status 2024-09-12T06:49:46.983 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_37.status 2024-09-12T06:49:46.983 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:49:46.983 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_37.status 2024-09-12T06:49:46.984 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_37 2024-09-12T06:49:46.984 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_37.status 2024-09-12T06:49:47.028 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 1 2024-09-12T06:49:47.028 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 1 2024-09-12T06:49:47.028 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:49:47.028 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6515], [object_number=6, tag_tid=1, entry_tid=6514]]], state=connected] 2024-09-12T06:49:47.028 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6515], [object_number=6, tag_tid=1, entry_tid=6514]]], state=connected] 2024-09-12T06:49:47.030 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_37.status 2024-09-12T06:49:47.032 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-09-12T06:49:47.033 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror image_37 2024-09-12T06:49:47.034 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:49:47.034 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:49:47.034 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_37 2024-09-12T06:49:47.034 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_37 '..*' 2024-09-12T06:49:47.034 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:49:47.034 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:49:47.034 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_37 2024-09-12T06:49:47.034 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:49:47.034 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_37.status 2024-09-12T06:49:47.034 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_37.status 2024-09-12T06:49:47.034 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:49:47.035 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_37.status 2024-09-12T06:49:47.035 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_37 2024-09-12T06:49:47.035 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_37.status 2024-09-12T06:49:47.079 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 1 2024-09-12T06:49:47.080 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 1 2024-09-12T06:49:47.080 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:49:47.080 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6515], [object_number=6, tag_tid=1, entry_tid=6514]]], state=connected] 2024-09-12T06:49:47.080 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6515], [object_number=6, tag_tid=1, entry_tid=6514]]], state=connected] 2024-09-12T06:49:47.082 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_37.status 2024-09-12T06:49:47.085 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-09-12T06:49:47.085 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-09-12T06:49:47.085 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:49:47.085 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_snap_present cluster1 mirror image_37 snap 2024-09-12T06:49:47.085 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:49:47.085 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:47.085 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_37 2024-09-12T06:49:47.085 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:49:47.085 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:49:47.085 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T06:49:47.085 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:49:48.085 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 info mirror/image_37@snap 2024-09-12T06:49:48.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:rbd image 'image_37': 2024-09-12T06:49:48.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: size 128 MiB in 32 objects 2024-09-12T06:49:48.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: order 22 (4 MiB objects) 2024-09-12T06:49:48.133 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: snapshot_count: 1 2024-09-12T06:49:48.133 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: id: 10a8dedb4e5e 2024-09-12T06:49:48.133 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: block_name_prefix: rbd_data.10a8dedb4e5e 2024-09-12T06:49:48.136 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: format: 2 2024-09-12T06:49:48.136 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: features: layering, exclusive-lock, journaling 2024-09-12T06:49:48.136 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: op_features: 2024-09-12T06:49:48.137 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: flags: 2024-09-12T06:49:48.137 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: create_timestamp: Thu Sep 12 06:43:34 2024 2024-09-12T06:49:48.137 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: access_timestamp: Thu Sep 12 06:43:34 2024 2024-09-12T06:49:48.137 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: modify_timestamp: Thu Sep 12 06:43:34 2024 2024-09-12T06:49:48.137 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: protected: False 2024-09-12T06:49:48.137 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: journal: 10a8dedb4e5e 2024-09-12T06:49:48.137 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring state: enabled 2024-09-12T06:49:48.137 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring mode: journal 2024-09-12T06:49:48.137 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring global id: af07d4e4-2267-489f-909c-dbba19f68a6b 2024-09-12T06:49:48.138 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring primary: false 2024-09-12T06:49:48.140 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:49:48.140 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror image_37 snap 2024-09-12T06:49:48.140 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:48.140 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_37 2024-09-12T06:49:48.140 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:49:48.141 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:49:48.141 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_37.export 2024-09-12T06:49:48.141 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_37.export 2024-09-12T06:49:48.141 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_37.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_37.export 2024-09-12T06:49:48.141 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export image_37@snap /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_37.export 2024-09-12T06:49:48.302 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:49:48.310 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export image_37@snap /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_37.export 2024-09-12T06:49:48.544 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:49:48.550 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_37.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_37.export 2024-09-12T06:49:48.605 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_37.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_37.export 2024-09-12T06:49:48.637 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:49:48.637 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:49:48.637 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_38 2024-09-12T06:49:48.637 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_snap cluster2 mirror image_38 snap 2024-09-12T06:49:48.637 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:49:48.637 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:48.637 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_38 2024-09-12T06:49:48.637 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:49:48.637 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror snap create image_38@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T06:49:49.548 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_started cluster1 mirror image_38 2024-09-12T06:49:49.548 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:49:49.548 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:49.548 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_38 2024-09-12T06:49:49.548 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_state cluster1 mirror image_38 started 2024-09-12T06:49:49.548 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:49:49.548 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:49.548 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_38 2024-09-12T06:49:49.548 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=started 2024-09-12T06:49:49.549 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:49:49.549 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:49:49.549 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:49:50.557 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_replay_state cluster1 mirror image_38 started 2024-09-12T06:49:50.557 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:49:50.557 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:50.557 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_38 2024-09-12T06:49:50.557 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=started 2024-09-12T06:49:50.557 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local status_result 2024-09-12T06:49:50.558 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=stopped 2024-09-12T06:49:50.558 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_38 2024-09-12T06:49:50.558 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -i state 2024-09-12T06:49:50.558 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_instance=cluster1 2024-09-12T06:49:50.558 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:49:50.559 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:49:50.559 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance=cluster1 2024-09-12T06:49:50.559 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local loop_instance 2024-09-12T06:49:50.559 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:49:50.559 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sleep 0 2024-09-12T06:49:50.559 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:49:50.559 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ seq 0 1 2024-09-12T06:49:50.559 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:49:50.559 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_38 2024-09-12T06:49:50.559 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:0 2024-09-12T06:49:50.559 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:49:50.559 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:49:50.560 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:49:50.560 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:49:50.560 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:49:50.560 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:49:50.560 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:49:50.560 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:49:50.560 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:49:50.560 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:49:50.560 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:49:50.560 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:49:50.560 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:49:50.560 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:49:50.560 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:0 2024-09-12T06:49:50.560 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:49:50.561 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:49:50.561 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:49:50.561 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:0 2024-09-12T06:49:50.561 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:49:50.561 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:49:50.561 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:49:50.561 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:49:50.561 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T06:49:50.561 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:49:50.561 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:49:50.561 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=0 2024-09-12T06:49:50.561 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=0 2024-09-12T06:49:50.561 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:49:50.572 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:50.573 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:50.573 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:50.573 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_38 2024-09-12T06:49:50.663 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2024-09-12T06:49:50.663 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T06:49:50.664 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:admin_socket: invalid command 2024-09-12T06:49:50.667 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:49:50.667 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_38 2024-09-12T06:49:50.668 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:1 2024-09-12T06:49:50.668 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:49:50.668 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:49:50.668 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T06:49:50.668 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:1 2024-09-12T06:49:50.668 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:49:50.668 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:49:50.668 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:49:50.668 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:49:50.668 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T06:49:50.668 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:49:50.668 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:49:50.668 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=1 2024-09-12T06:49:50.669 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:49:50.669 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T06:49:50.669 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:1 2024-09-12T06:49:50.669 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:49:50.669 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:49:50.669 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T06:49:50.669 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:1 2024-09-12T06:49:50.669 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:49:50.669 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:49:50.669 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:49:50.669 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:49:50.669 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T06:49:50.669 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:49:50.670 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:49:50.670 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=1 2024-09-12T06:49:50.670 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=1 2024-09-12T06:49:50.670 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T06:49:50.686 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:49:50.686 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:49:50.686 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:49:50.686 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_38 2024-09-12T06:49:50.784 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ return 0 2024-09-12T06:49:50.784 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T06:49:50.785 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo ' "state": "Replaying"' 2024-09-12T06:49:50.785 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep -i Replaying 2024-09-12T06:49:50.786 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: "state": "Replaying" 2024-09-12T06:49:50.790 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_state=started 2024-09-12T06:49:50.791 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test started = started 2024-09-12T06:49:50.791 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:49:50.791 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_38 2024-09-12T06:49:50.791 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:49:50.791 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:49:50.792 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:49:50.792 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:49:50.792 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_38 2024-09-12T06:49:50.792 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' journal = journal ']' 2024-09-12T06:49:50.793 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror image_38 2024-09-12T06:49:50.793 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:49:50.793 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:49:50.793 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:49:50.793 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:49:50.793 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_38 2024-09-12T06:49:50.793 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-12T06:49:50.794 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-12T06:49:50.794 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:49:50.794 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:49:50.794 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:49:50.987 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror image_38 2024-09-12T06:49:50.987 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:49:50.988 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:50.988 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_38 2024-09-12T06:49:50.988 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:49:50.988 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n image_38 ']' 2024-09-12T06:49:50.988 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/image_38' 2024-09-12T06:49:50.988 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_38 2024-09-12T06:49:50.988 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:49:50.988 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:49:50.988 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:49:50.988 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:49:50.988 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:49:50.988 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:49:50.989 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:49:50.989 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:49:50.989 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:49:50.991 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:49:50.991 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_38 2024-09-12T06:49:50.991 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:49:50.992 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:49:50.992 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:49:50.992 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:49:50.992 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:49:50.992 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:49:50.992 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:49:50.992 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:49:50.992 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:49:50.992 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:49:50.992 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:49:50.992 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:49:50.992 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:49:50.993 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:49:50.993 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:49:50.993 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:49:50.993 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:49:50.993 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:49:50.993 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:49:50.993 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:49:50.993 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:49:50.993 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:49:50.993 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:49:50.993 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:49:50.993 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:49:50.994 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:49:50.994 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:49:50.994 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:49:50.994 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:49:50.994 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:49:51.009 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:51.009 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:51.009 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:51.009 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_38 2024-09-12T06:49:51.095 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2024-09-12T06:49:51.095 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd mirror flush : flush rbd mirror 2024-09-12T06:49:51.095 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:admin_socket: invalid command 2024-09-12T06:49:51.098 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:49:51.098 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_38 2024-09-12T06:49:51.098 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:1 2024-09-12T06:49:51.099 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:49:51.099 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:49:51.099 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-12T06:49:51.099 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:1 2024-09-12T06:49:51.099 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:49:51.099 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:49:51.099 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:49:51.099 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=1 2024-09-12T06:49:51.099 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-12T06:49:51.099 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:49:51.099 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:49:51.099 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=1 2024-09-12T06:49:51.099 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:49:51.099 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-12T06:49:51.099 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:1 2024-09-12T06:49:51.100 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:49:51.100 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:49:51.100 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T06:49:51.100 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:1 2024-09-12T06:49:51.100 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:49:51.100 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:49:51.100 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:49:51.100 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:49:51.100 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T06:49:51.100 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:49:51.100 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:49:51.100 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=1 2024-09-12T06:49:51.100 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:49:51.100 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T06:49:51.117 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:49:51.117 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:49:51.117 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:49:51.117 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_38 2024-09-12T06:49:51.217 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:49:51.217 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror image_38 2024-09-12T06:49:51.217 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:49:51.218 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:49:51.218 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_38 2024-09-12T06:49:51.218 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_38 '' 2024-09-12T06:49:51.218 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:49:51.218 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:49:51.218 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_38 2024-09-12T06:49:51.218 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:49:51.218 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_38.status 2024-09-12T06:49:51.218 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_38.status 2024-09-12T06:49:51.218 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:49:51.219 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_38.status 2024-09-12T06:49:51.219 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_38 2024-09-12T06:49:51.219 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_38.status 2024-09-12T06:49:51.265 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 1 2024-09-12T06:49:51.265 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 1 2024-09-12T06:49:51.265 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:49:51.265 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5187], [object_number=6, tag_tid=1, entry_tid=5186]]], state=connected] 2024-09-12T06:49:51.265 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5187], [object_number=6, tag_tid=1, entry_tid=5186]]], state=connected] 2024-09-12T06:49:51.267 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_38.status 2024-09-12T06:49:51.270 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-12T06:49:51.271 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror image_38 2024-09-12T06:49:51.271 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:49:51.271 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:49:51.271 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_38 2024-09-12T06:49:51.271 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_38 '..*' 2024-09-12T06:49:51.272 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:49:51.272 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:49:51.272 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_38 2024-09-12T06:49:51.272 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:49:51.272 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_38.status 2024-09-12T06:49:51.272 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_38.status 2024-09-12T06:49:51.273 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:49:51.273 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_38.status 2024-09-12T06:49:51.273 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_38 2024-09-12T06:49:51.274 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_38.status 2024-09-12T06:49:51.318 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 1 2024-09-12T06:49:51.318 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 1 2024-09-12T06:49:51.318 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:49:51.319 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5187], [object_number=6, tag_tid=1, entry_tid=5186]]], state=connected] 2024-09-12T06:49:51.319 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5187], [object_number=6, tag_tid=1, entry_tid=5186]]], state=connected] 2024-09-12T06:49:51.321 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_38.status 2024-09-12T06:49:51.323 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-12T06:49:51.323 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-09-12T06:49:51.323 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:49:51.323 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_snap_present cluster1 mirror image_38 snap 2024-09-12T06:49:51.324 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:49:51.324 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:51.324 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_38 2024-09-12T06:49:51.324 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:49:51.324 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:49:51.324 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T06:49:51.324 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:49:52.324 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 info mirror/image_38@snap 2024-09-12T06:49:52.371 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:rbd image 'image_38': 2024-09-12T06:49:52.371 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: size 128 MiB in 32 objects 2024-09-12T06:49:52.371 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: order 22 (4 MiB objects) 2024-09-12T06:49:52.371 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: snapshot_count: 1 2024-09-12T06:49:52.372 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: id: 10af9517193a 2024-09-12T06:49:52.372 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: block_name_prefix: rbd_data.10af9517193a 2024-09-12T06:49:52.372 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: format: 2 2024-09-12T06:49:52.372 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: features: layering, exclusive-lock, journaling 2024-09-12T06:49:52.372 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: op_features: 2024-09-12T06:49:52.372 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: flags: 2024-09-12T06:49:52.372 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: create_timestamp: Thu Sep 12 06:43:43 2024 2024-09-12T06:49:52.372 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: access_timestamp: Thu Sep 12 06:43:43 2024 2024-09-12T06:49:52.372 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: modify_timestamp: Thu Sep 12 06:44:44 2024 2024-09-12T06:49:52.372 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: protected: False 2024-09-12T06:49:52.372 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: journal: 10af9517193a 2024-09-12T06:49:52.372 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring state: enabled 2024-09-12T06:49:52.372 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring mode: journal 2024-09-12T06:49:52.373 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring global id: 0946308e-64a5-4451-9e2a-ce9adc0ea909 2024-09-12T06:49:52.373 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring primary: false 2024-09-12T06:49:52.375 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:49:52.375 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror image_38 snap 2024-09-12T06:49:52.375 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:52.375 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_38 2024-09-12T06:49:52.375 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:49:52.375 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:49:52.375 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_38.export 2024-09-12T06:49:52.376 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_38.export 2024-09-12T06:49:52.376 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_38.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_38.export 2024-09-12T06:49:52.376 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export image_38@snap /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_38.export 2024-09-12T06:49:52.529 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:49:52.537 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export image_38@snap /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_38.export 2024-09-12T06:49:52.718 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:49:52.724 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_38.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_38.export 2024-09-12T06:49:52.780 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_38.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_38.export 2024-09-12T06:49:52.811 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:49:52.811 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:49:52.811 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_39 2024-09-12T06:49:52.811 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_snap cluster2 mirror image_39 snap 2024-09-12T06:49:52.811 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:49:52.811 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:52.811 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_39 2024-09-12T06:49:52.812 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:49:52.812 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror snap create image_39@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T06:49:54.555 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_started cluster1 mirror image_39 2024-09-12T06:49:54.556 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:49:54.556 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:54.556 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_39 2024-09-12T06:49:54.556 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_state cluster1 mirror image_39 started 2024-09-12T06:49:54.556 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:49:54.556 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:54.556 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_39 2024-09-12T06:49:54.556 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=started 2024-09-12T06:49:54.556 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:49:54.556 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:49:54.556 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:49:54.906 INFO:tasks.ceph.cluster2.osd.0.smithi152.stderr:2024-09-12T06:49:54.908+0000 7fd61689f640 -1 reset not still connected to 0x55e763645930 2024-09-12T06:49:54.908 INFO:tasks.ceph.cluster2.osd.0.smithi152.stderr:2024-09-12T06:49:54.909+0000 7fd61689f640 -1 reset not still connected to 0x55e764eca4e0 2024-09-12T06:49:54.908 INFO:tasks.ceph.cluster2.osd.0.smithi152.stderr:2024-09-12T06:49:54.909+0000 7fd61689f640 -1 reset not still connected to 0x55e7657f0750 2024-09-12T06:49:54.910 INFO:tasks.ceph.cluster2.osd.0.smithi152.stderr:2024-09-12T06:49:54.912+0000 7fd61689f640 -1 reset not still connected to 0x55e76d1eba00 2024-09-12T06:49:54.910 INFO:tasks.ceph.cluster2.osd.0.smithi152.stderr:2024-09-12T06:49:54.912+0000 7fd61689f640 -1 reset not still connected to 0x55e779472270 2024-09-12T06:49:54.910 INFO:tasks.ceph.cluster2.osd.0.smithi152.stderr:2024-09-12T06:49:54.912+0000 7fd61689f640 -1 reset not still connected to 0x55e782e41ee0 2024-09-12T06:49:54.910 INFO:tasks.ceph.cluster2.osd.0.smithi152.stderr:2024-09-12T06:49:54.912+0000 7fd61689f640 -1 reset not still connected to 0x55e785214270 2024-09-12T06:49:54.910 INFO:tasks.ceph.cluster2.osd.0.smithi152.stderr:2024-09-12T06:49:54.912+0000 7fd61689f640 -1 reset not still connected to 0x55e785c50b60 2024-09-12T06:49:54.911 INFO:tasks.ceph.cluster2.osd.0.smithi152.stderr:2024-09-12T06:49:54.912+0000 7fd61689f640 -1 reset not still connected to 0x55e785c51520 2024-09-12T06:49:54.911 INFO:tasks.ceph.cluster2.osd.0.smithi152.stderr:2024-09-12T06:49:54.912+0000 7fd61689f640 -1 reset not still connected to 0x55e78805cd00 2024-09-12T06:49:55.557 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_replay_state cluster1 mirror image_39 started 2024-09-12T06:49:55.557 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:49:55.558 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:55.558 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_39 2024-09-12T06:49:55.558 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=started 2024-09-12T06:49:55.558 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local status_result 2024-09-12T06:49:55.558 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=stopped 2024-09-12T06:49:55.558 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_39 2024-09-12T06:49:55.558 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_instance=cluster1 2024-09-12T06:49:55.558 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:49:55.558 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -i state 2024-09-12T06:49:55.558 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:49:55.558 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance=cluster1 2024-09-12T06:49:55.558 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local loop_instance 2024-09-12T06:49:55.558 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:49:55.558 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sleep 0 2024-09-12T06:49:55.562 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:49:55.562 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ seq 0 1 2024-09-12T06:49:55.562 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:49:55.562 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_39 2024-09-12T06:49:55.562 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:0 2024-09-12T06:49:55.562 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:49:55.562 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:49:55.562 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:49:55.562 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:49:55.562 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:49:55.563 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:49:55.563 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:49:55.563 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:49:55.563 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:49:55.563 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:49:55.563 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:49:55.563 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:49:55.563 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:49:55.563 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:49:55.563 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:0 2024-09-12T06:49:55.563 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:49:55.563 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:49:55.563 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:49:55.563 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:0 2024-09-12T06:49:55.564 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:49:55.564 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:49:55.564 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:49:55.564 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:49:55.564 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T06:49:55.564 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:49:55.564 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:49:55.564 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=0 2024-09-12T06:49:55.564 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=0 2024-09-12T06:49:55.564 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:49:55.580 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:55.580 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:55.580 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:55.580 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_39 2024-09-12T06:49:55.678 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ return 0 2024-09-12T06:49:55.678 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T06:49:55.679 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo ' "state": "Replaying"' 2024-09-12T06:49:55.679 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep -i Replaying 2024-09-12T06:49:55.679 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: "state": "Replaying" 2024-09-12T06:49:55.681 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_state=started 2024-09-12T06:49:55.681 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test started = started 2024-09-12T06:49:55.681 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:49:55.681 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_39 2024-09-12T06:49:55.681 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:49:55.681 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:49:55.681 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:49:55.681 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:49:55.681 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_39 2024-09-12T06:49:55.682 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' journal = journal ']' 2024-09-12T06:49:55.682 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror image_39 2024-09-12T06:49:55.682 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:49:55.682 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:49:55.682 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:49:55.682 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:49:55.682 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_39 2024-09-12T06:49:55.682 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-12T06:49:55.682 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-12T06:49:55.682 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:49:55.682 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:49:55.682 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:49:55.882 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror image_39 2024-09-12T06:49:55.882 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:49:55.882 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:55.882 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_39 2024-09-12T06:49:55.882 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:49:55.882 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n image_39 ']' 2024-09-12T06:49:55.882 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/image_39' 2024-09-12T06:49:55.882 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_39 2024-09-12T06:49:55.883 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:49:55.883 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:49:55.883 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:49:55.883 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:49:55.883 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:49:55.883 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:49:55.883 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:49:55.883 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:49:55.883 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:49:55.886 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:49:55.886 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_39 2024-09-12T06:49:55.886 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:49:55.887 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:49:55.887 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:49:55.887 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:49:55.887 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:49:55.887 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:49:55.887 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:49:55.887 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:49:55.887 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:49:55.887 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:49:55.887 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:49:55.887 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:49:55.887 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:49:55.887 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:49:55.887 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:49:55.888 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:49:55.888 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:49:55.888 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:49:55.888 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:49:55.888 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:49:55.888 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:49:55.888 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:49:55.888 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:49:55.888 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:49:55.888 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:49:55.889 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:49:55.889 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:49:55.889 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:49:55.889 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:49:55.889 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:49:55.903 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:55.903 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:55.904 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:49:55.904 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_39 2024-09-12T06:49:56.003 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:49:56.003 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror image_39 2024-09-12T06:49:56.003 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:49:56.003 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:49:56.003 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_39 2024-09-12T06:49:56.003 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_39 '' 2024-09-12T06:49:56.003 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:49:56.003 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:49:56.003 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_39 2024-09-12T06:49:56.003 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:49:56.003 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_39.status 2024-09-12T06:49:56.003 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_39.status 2024-09-12T06:49:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:49:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_39.status 2024-09-12T06:49:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_39 2024-09-12T06:49:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_39.status 2024-09-12T06:49:56.050 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 3 2024-09-12T06:49:56.050 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 3 2024-09-12T06:49:56.050 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:49:56.050 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=12643], [object_number=14, tag_tid=1, entry_tid=12642]]], state=connected] 2024-09-12T06:49:56.050 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=12643], [object_number=14, tag_tid=1, entry_tid=12642]]], state=connected] 2024-09-12T06:49:56.052 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_39.status 2024-09-12T06:49:56.054 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-09-12T06:49:56.055 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror image_39 2024-09-12T06:49:56.055 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:49:56.055 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:49:56.055 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_39 2024-09-12T06:49:56.055 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_39 '..*' 2024-09-12T06:49:56.056 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:49:56.056 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:49:56.056 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_39 2024-09-12T06:49:56.056 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:49:56.056 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_39.status 2024-09-12T06:49:56.056 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_39.status 2024-09-12T06:49:56.056 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:49:56.056 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_39.status 2024-09-12T06:49:56.057 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_39 2024-09-12T06:49:56.057 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_39.status 2024-09-12T06:49:56.101 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 3 2024-09-12T06:49:56.102 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 3 2024-09-12T06:49:56.102 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:49:56.102 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=12643], [object_number=14, tag_tid=1, entry_tid=12642]]], state=connected] 2024-09-12T06:49:56.102 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=12643], [object_number=14, tag_tid=1, entry_tid=12642]]], state=connected] 2024-09-12T06:49:56.104 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_39.status 2024-09-12T06:49:56.106 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-09-12T06:49:56.107 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=1' 2024-09-12T06:49:56.107 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:49:56.107 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_snap_present cluster1 mirror image_39 snap 2024-09-12T06:49:56.107 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:49:56.107 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:56.107 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_39 2024-09-12T06:49:56.107 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:49:56.107 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:49:56.107 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T06:49:56.107 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:49:57.107 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 info mirror/image_39@snap 2024-09-12T06:49:57.155 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:rbd image 'image_39': 2024-09-12T06:49:57.155 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: size 128 MiB in 32 objects 2024-09-12T06:49:57.155 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: order 22 (4 MiB objects) 2024-09-12T06:49:57.155 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: snapshot_count: 1 2024-09-12T06:49:57.155 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: id: 10a865154803 2024-09-12T06:49:57.155 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: block_name_prefix: rbd_data.10a865154803 2024-09-12T06:49:57.156 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: format: 2 2024-09-12T06:49:57.156 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: features: layering, exclusive-lock, journaling 2024-09-12T06:49:57.156 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: op_features: 2024-09-12T06:49:57.156 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: flags: 2024-09-12T06:49:57.156 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: create_timestamp: Thu Sep 12 06:43:46 2024 2024-09-12T06:49:57.156 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: access_timestamp: Thu Sep 12 06:43:46 2024 2024-09-12T06:49:57.156 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: modify_timestamp: Thu Sep 12 06:44:47 2024 2024-09-12T06:49:57.156 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: protected: False 2024-09-12T06:49:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: journal: 10a865154803 2024-09-12T06:49:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring state: enabled 2024-09-12T06:49:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring mode: journal 2024-09-12T06:49:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring global id: c5458b25-1669-4f44-bb1d-45e10a9fbc20 2024-09-12T06:49:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring primary: false 2024-09-12T06:49:57.159 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:49:57.159 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror image_39 snap 2024-09-12T06:49:57.159 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:57.159 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_39 2024-09-12T06:49:57.159 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:49:57.159 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:49:57.159 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_39.export 2024-09-12T06:49:57.159 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_39.export 2024-09-12T06:49:57.159 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_39.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_39.export 2024-09-12T06:49:57.159 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export image_39@snap /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_39.export 2024-09-12T06:49:57.316 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:49:57.325 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export image_39@snap /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_39.export 2024-09-12T06:49:57.456 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:49:57.458+0000 7fa3136a9640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-09-12T06:49:57.553 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:49:57.558 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_39.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_39.export 2024-09-12T06:49:57.614 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_39.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_39.export 2024-09-12T06:49:57.646 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:49:57.646 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:49:57.646 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_40 2024-09-12T06:49:57.646 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_snap cluster2 mirror image_40 snap 2024-09-12T06:49:57.646 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:49:57.646 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:49:57.646 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_40 2024-09-12T06:49:57.646 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:49:57.646 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror snap create image_40@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T06:49:58.634 INFO:tasks.ceph.cluster2.osd.2.smithi152.stderr:2024-09-12T06:49:58.636+0000 7f47757b1640 -1 bdev(0x56163c600c00 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 4 2024-09-12T06:49:58.635 INFO:tasks.ceph.cluster2.osd.2.smithi152.stderr:2024-09-12T06:49:58.636+0000 7f4777fb6640 -1 bdev(0x56163c600c00 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 4 2024-09-12T06:49:59.818 INFO:tasks.ceph.cluster2.osd.2.smithi152.stderr:2024-09-12T06:49:59.820+0000 7f4789beb640 -1 reset not still connected to 0x56163d519e10 2024-09-12T06:49:59.819 INFO:tasks.ceph.cluster2.osd.2.smithi152.stderr:2024-09-12T06:49:59.820+0000 7f4789beb640 -1 reset not still connected to 0x56163df360d0 2024-09-12T06:49:59.819 INFO:tasks.ceph.cluster2.osd.2.smithi152.stderr:2024-09-12T06:49:59.820+0000 7f4789beb640 -1 reset not still connected to 0x56163df36270 2024-09-12T06:50:00.632 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_started cluster1 mirror image_40 2024-09-12T06:50:00.632 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:50:00.632 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:00.632 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_40 2024-09-12T06:50:00.632 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_state cluster1 mirror image_40 started 2024-09-12T06:50:00.632 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:50:00.632 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:00.632 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_40 2024-09-12T06:50:00.633 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=started 2024-09-12T06:50:00.633 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:50:00.633 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:50:00.633 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:50:01.633 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_replay_state cluster1 mirror image_40 started 2024-09-12T06:50:01.633 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:50:01.634 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:01.634 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_40 2024-09-12T06:50:01.634 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=started 2024-09-12T06:50:01.634 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local status_result 2024-09-12T06:50:01.634 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=stopped 2024-09-12T06:50:01.634 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_40 2024-09-12T06:50:01.634 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -i state 2024-09-12T06:50:01.634 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_instance=cluster1 2024-09-12T06:50:01.634 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:50:01.634 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:50:01.634 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance=cluster1 2024-09-12T06:50:01.634 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local loop_instance 2024-09-12T06:50:01.634 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:50:01.635 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sleep 0 2024-09-12T06:50:01.637 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:50:01.637 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ seq 0 1 2024-09-12T06:50:01.638 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:50:01.638 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_40 2024-09-12T06:50:01.638 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:0 2024-09-12T06:50:01.638 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:50:01.638 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:50:01.638 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:50:01.638 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:50:01.638 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:50:01.638 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:50:01.638 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:50:01.638 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:50:01.638 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:50:01.638 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:50:01.638 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:50:01.639 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:50:01.639 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:50:01.639 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:50:01.639 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:0 2024-09-12T06:50:01.639 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:50:01.639 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:50:01.639 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:50:01.639 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:0 2024-09-12T06:50:01.639 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:50:01.639 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:50:01.639 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:50:01.639 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:50:01.639 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T06:50:01.639 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:50:01.639 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:50:01.640 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=0 2024-09-12T06:50:01.640 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=0 2024-09-12T06:50:01.640 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:50:01.656 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:50:01.656 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:50:01.656 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:50:01.656 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_40 2024-09-12T06:50:01.748 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2024-09-12T06:50:01.749 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T06:50:01.749 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:admin_socket: invalid command 2024-09-12T06:50:01.752 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:50:01.752 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_40 2024-09-12T06:50:01.752 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:1 2024-09-12T06:50:01.752 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:50:01.753 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:50:01.753 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T06:50:01.753 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:1 2024-09-12T06:50:01.753 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:50:01.753 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:50:01.753 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:50:01.753 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:50:01.753 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T06:50:01.753 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:50:01.753 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:50:01.753 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=1 2024-09-12T06:50:01.754 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:50:01.754 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T06:50:01.754 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:1 2024-09-12T06:50:01.754 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:50:01.754 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:50:01.754 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T06:50:01.754 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:1 2024-09-12T06:50:01.754 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:50:01.754 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:50:01.754 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:50:01.754 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:50:01.754 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T06:50:01.754 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:50:01.755 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:50:01.755 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=1 2024-09-12T06:50:01.755 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=1 2024-09-12T06:50:01.755 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T06:50:01.771 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:50:01.772 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:50:01.772 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:50:01.772 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_40 2024-09-12T06:50:01.870 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ return 0 2024-09-12T06:50:01.870 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T06:50:01.871 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo ' "state": "Replaying"' 2024-09-12T06:50:01.871 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep -i Replaying 2024-09-12T06:50:01.871 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: "state": "Replaying" 2024-09-12T06:50:01.873 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_state=started 2024-09-12T06:50:01.873 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test started = started 2024-09-12T06:50:01.873 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:50:01.874 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_40 2024-09-12T06:50:01.874 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:50:01.874 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:50:01.874 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:50:01.874 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:50:01.874 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_40 2024-09-12T06:50:01.874 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' journal = journal ']' 2024-09-12T06:50:01.874 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror image_40 2024-09-12T06:50:01.874 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:50:01.874 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:50:01.874 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:50:01.874 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:50:01.874 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_40 2024-09-12T06:50:01.874 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-12T06:50:01.875 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-12T06:50:01.875 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:50:01.875 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:50:01.875 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:50:02.074 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror image_40 2024-09-12T06:50:02.075 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:50:02.075 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:02.075 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_40 2024-09-12T06:50:02.075 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:50:02.075 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n image_40 ']' 2024-09-12T06:50:02.075 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/image_40' 2024-09-12T06:50:02.076 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_40 2024-09-12T06:50:02.076 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:50:02.076 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:50:02.076 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:50:02.076 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:50:02.076 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:50:02.076 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:50:02.077 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:50:02.077 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:50:02.077 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:50:02.081 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:50:02.081 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_40 2024-09-12T06:50:02.081 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:50:02.081 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:50:02.082 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:50:02.082 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:50:02.082 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:50:02.082 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:50:02.082 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:50:02.083 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:50:02.083 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:50:02.083 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:50:02.083 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:50:02.083 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:50:02.083 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:50:02.084 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:50:02.084 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:50:02.084 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:50:02.084 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:50:02.084 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:50:02.084 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:50:02.085 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:50:02.085 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:50:02.085 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:50:02.086 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:50:02.086 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:50:02.086 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:50:02.086 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:50:02.086 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:50:02.086 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:50:02.087 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:50:02.087 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:50:02.096 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:50:02.097 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:50:02.097 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:50:02.097 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_40 2024-09-12T06:50:02.182 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2024-09-12T06:50:02.183 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd mirror flush : flush rbd mirror 2024-09-12T06:50:02.183 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:admin_socket: invalid command 2024-09-12T06:50:02.186 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:50:02.186 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_40 2024-09-12T06:50:02.186 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:1 2024-09-12T06:50:02.186 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:50:02.187 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:50:02.187 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-12T06:50:02.187 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:1 2024-09-12T06:50:02.187 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:50:02.187 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:50:02.187 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:50:02.187 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=1 2024-09-12T06:50:02.187 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-12T06:50:02.187 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:50:02.187 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:50:02.187 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=1 2024-09-12T06:50:02.188 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:50:02.188 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-12T06:50:02.188 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:1 2024-09-12T06:50:02.188 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:50:02.188 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:50:02.188 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T06:50:02.188 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:1 2024-09-12T06:50:02.188 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:50:02.188 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:50:02.188 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:50:02.188 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:50:02.188 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T06:50:02.188 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:50:02.189 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:50:02.189 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=1 2024-09-12T06:50:02.189 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:50:02.189 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T06:50:02.205 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:50:02.205 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:50:02.205 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:50:02.205 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_40 2024-09-12T06:50:02.305 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:50:02.305 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror image_40 2024-09-12T06:50:02.305 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:50:02.305 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:50:02.305 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_40 2024-09-12T06:50:02.305 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_40 '' 2024-09-12T06:50:02.305 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:50:02.305 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:50:02.305 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_40 2024-09-12T06:50:02.305 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:50:02.306 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_40.status 2024-09-12T06:50:02.306 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_40.status 2024-09-12T06:50:02.306 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:50:02.306 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_40.status 2024-09-12T06:50:02.306 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_40 2024-09-12T06:50:02.306 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_40.status 2024-09-12T06:50:02.353 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 2 2024-09-12T06:50:02.353 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 2 2024-09-12T06:50:02.353 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:50:02.353 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=7931], [object_number=10, tag_tid=1, entry_tid=7930]]], state=connected] 2024-09-12T06:50:02.353 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=7931], [object_number=10, tag_tid=1, entry_tid=7930]]], state=connected] 2024-09-12T06:50:02.356 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_40.status 2024-09-12T06:50:02.357 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-09-12T06:50:02.359 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror image_40 2024-09-12T06:50:02.359 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:50:02.359 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:50:02.359 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_40 2024-09-12T06:50:02.359 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_40 '..*' 2024-09-12T06:50:02.359 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:50:02.359 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:50:02.359 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_40 2024-09-12T06:50:02.359 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:50:02.360 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_40.status 2024-09-12T06:50:02.360 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_40.status 2024-09-12T06:50:02.360 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:50:02.360 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_40.status 2024-09-12T06:50:02.360 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_40 2024-09-12T06:50:02.360 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_40.status 2024-09-12T06:50:02.405 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 2 2024-09-12T06:50:02.405 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 2 2024-09-12T06:50:02.405 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:50:02.405 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=7931], [object_number=10, tag_tid=1, entry_tid=7930]]], state=connected] 2024-09-12T06:50:02.406 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=7931], [object_number=10, tag_tid=1, entry_tid=7930]]], state=connected] 2024-09-12T06:50:02.408 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_40.status 2024-09-12T06:50:02.411 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-09-12T06:50:02.411 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-09-12T06:50:02.411 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:50:02.411 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_snap_present cluster1 mirror image_40 snap 2024-09-12T06:50:02.411 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:50:02.411 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:02.411 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_40 2024-09-12T06:50:02.411 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:50:02.412 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:50:02.412 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T06:50:02.412 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:50:03.411 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 info mirror/image_40@snap 2024-09-12T06:50:03.459 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:rbd image 'image_40': 2024-09-12T06:50:03.459 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: size 128 MiB in 32 objects 2024-09-12T06:50:03.459 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: order 22 (4 MiB objects) 2024-09-12T06:50:03.459 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: snapshot_count: 1 2024-09-12T06:50:03.459 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: id: 10aff23de939 2024-09-12T06:50:03.459 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: block_name_prefix: rbd_data.10aff23de939 2024-09-12T06:50:03.459 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: format: 2 2024-09-12T06:50:03.459 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: features: layering, exclusive-lock, journaling 2024-09-12T06:50:03.460 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: op_features: 2024-09-12T06:50:03.460 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: flags: 2024-09-12T06:50:03.460 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: create_timestamp: Thu Sep 12 06:44:02 2024 2024-09-12T06:50:03.460 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: access_timestamp: Thu Sep 12 06:44:02 2024 2024-09-12T06:50:03.460 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: modify_timestamp: Thu Sep 12 06:45:04 2024 2024-09-12T06:50:03.460 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: protected: False 2024-09-12T06:50:03.460 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: journal: 10aff23de939 2024-09-12T06:50:03.460 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring state: enabled 2024-09-12T06:50:03.460 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring mode: journal 2024-09-12T06:50:03.460 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring global id: 144d8d61-7ac8-483f-8923-df10ac7d61e6 2024-09-12T06:50:03.460 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring primary: false 2024-09-12T06:50:03.463 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:50:03.463 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror image_40 snap 2024-09-12T06:50:03.463 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:03.463 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_40 2024-09-12T06:50:03.463 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:50:03.463 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:50:03.463 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_40.export 2024-09-12T06:50:03.463 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_40.export 2024-09-12T06:50:03.463 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_40.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_40.export 2024-09-12T06:50:03.464 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export image_40@snap /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_40.export 2024-09-12T06:50:03.615 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:50:03.621 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export image_40@snap /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_40.export 2024-09-12T06:50:03.807 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:50:03.815 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_40.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_40.export 2024-09-12T06:50:03.870 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_40.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_40.export 2024-09-12T06:50:03.902 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:50:03.902 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:50:03.902 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_41 2024-09-12T06:50:03.903 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_snap cluster2 mirror image_41 snap 2024-09-12T06:50:03.903 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:50:03.903 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:03.903 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_41 2024-09-12T06:50:03.903 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:50:03.903 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror snap create image_41@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T06:50:04.635 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_started cluster1 mirror image_41 2024-09-12T06:50:04.635 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:50:04.635 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:04.636 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_41 2024-09-12T06:50:04.636 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_state cluster1 mirror image_41 started 2024-09-12T06:50:04.636 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:50:04.636 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:04.636 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_41 2024-09-12T06:50:04.636 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=started 2024-09-12T06:50:04.636 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:50:04.636 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:50:04.636 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:50:05.637 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_replay_state cluster1 mirror image_41 started 2024-09-12T06:50:05.638 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:50:05.638 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:05.638 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_41 2024-09-12T06:50:05.638 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=started 2024-09-12T06:50:05.638 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local status_result 2024-09-12T06:50:05.638 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=stopped 2024-09-12T06:50:05.638 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_41 2024-09-12T06:50:05.638 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -i state 2024-09-12T06:50:05.638 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_instance=cluster1 2024-09-12T06:50:05.638 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:50:05.638 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:50:05.638 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance=cluster1 2024-09-12T06:50:05.639 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local loop_instance 2024-09-12T06:50:05.639 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:50:05.639 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sleep 0 2024-09-12T06:50:05.642 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:50:05.642 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ seq 0 1 2024-09-12T06:50:05.642 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:50:05.642 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_41 2024-09-12T06:50:05.642 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:0 2024-09-12T06:50:05.642 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:50:05.643 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:50:05.643 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:50:05.643 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:50:05.643 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:50:05.643 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:50:05.643 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:50:05.643 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:50:05.643 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:50:05.643 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:50:05.643 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:50:05.643 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:50:05.643 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:50:05.643 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:50:05.643 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:0 2024-09-12T06:50:05.644 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:50:05.644 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:50:05.644 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:50:05.644 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:0 2024-09-12T06:50:05.644 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:50:05.644 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:50:05.644 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:50:05.644 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:50:05.644 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T06:50:05.644 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:50:05.644 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:50:05.644 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=0 2024-09-12T06:50:05.644 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=0 2024-09-12T06:50:05.644 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:50:05.663 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:50:05.663 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:50:05.663 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:50:05.663 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_41 2024-09-12T06:50:05.768 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ return 0 2024-09-12T06:50:05.769 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T06:50:05.769 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo ' "state": "Replaying"' 2024-09-12T06:50:05.769 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep -i Replaying 2024-09-12T06:50:05.770 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: "state": "Replaying" 2024-09-12T06:50:05.772 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_state=started 2024-09-12T06:50:05.772 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test started = started 2024-09-12T06:50:05.772 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:50:05.772 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_41 2024-09-12T06:50:05.772 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:50:05.772 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:50:05.772 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:50:05.772 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:50:05.772 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_41 2024-09-12T06:50:05.772 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' journal = journal ']' 2024-09-12T06:50:05.773 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror image_41 2024-09-12T06:50:05.773 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:50:05.773 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:50:05.773 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:50:05.773 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:50:05.773 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_41 2024-09-12T06:50:05.773 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-12T06:50:05.773 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-12T06:50:05.773 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:50:05.773 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:50:05.773 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:50:05.973 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror image_41 2024-09-12T06:50:05.973 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:50:05.974 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:05.974 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_41 2024-09-12T06:50:05.974 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:50:05.974 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n image_41 ']' 2024-09-12T06:50:05.974 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/image_41' 2024-09-12T06:50:05.974 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_41 2024-09-12T06:50:05.974 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:50:05.974 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:50:05.974 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:50:05.974 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:50:05.974 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:50:05.975 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:50:05.975 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:50:05.975 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:50:05.975 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:50:05.978 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:50:05.978 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_41 2024-09-12T06:50:05.978 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:50:05.978 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:50:05.978 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:50:05.978 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:50:05.978 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:50:05.979 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:50:05.979 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:50:05.979 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:50:05.979 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:50:05.979 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:50:05.979 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:50:05.979 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:50:05.979 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:50:05.979 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:50:05.979 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:50:05.979 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:50:05.979 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:50:05.979 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:50:05.979 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:50:05.980 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:50:05.980 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:50:05.980 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:50:05.980 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:50:05.980 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:50:05.980 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:50:05.980 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:50:05.980 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:50:05.980 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:50:05.980 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:50:05.980 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:50:05.995 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:50:05.995 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:50:05.995 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:50:05.995 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_41 2024-09-12T06:50:06.093 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:50:06.094 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror image_41 2024-09-12T06:50:06.094 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:50:06.094 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:50:06.094 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_41 2024-09-12T06:50:06.094 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_41 '' 2024-09-12T06:50:06.094 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:50:06.094 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:50:06.094 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_41 2024-09-12T06:50:06.094 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:50:06.094 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_41.status 2024-09-12T06:50:06.094 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_41.status 2024-09-12T06:50:06.094 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:50:06.095 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_41.status 2024-09-12T06:50:06.095 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_41 2024-09-12T06:50:06.095 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_41.status 2024-09-12T06:50:06.142 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 1 2024-09-12T06:50:06.143 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 1 2024-09-12T06:50:06.143 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:50:06.143 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=3630], [object_number=7, tag_tid=1, entry_tid=3627]]], state=connected] 2024-09-12T06:50:06.143 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=3630], [object_number=7, tag_tid=1, entry_tid=3627]]], state=connected] 2024-09-12T06:50:06.145 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_41.status 2024-09-12T06:50:06.147 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-12T06:50:06.149 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror image_41 2024-09-12T06:50:06.149 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:50:06.149 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:50:06.149 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_41 2024-09-12T06:50:06.149 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_41 '..*' 2024-09-12T06:50:06.149 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:50:06.149 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:50:06.149 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_41 2024-09-12T06:50:06.149 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:50:06.149 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_41.status 2024-09-12T06:50:06.149 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_41.status 2024-09-12T06:50:06.149 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:50:06.150 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_41.status 2024-09-12T06:50:06.150 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_41 2024-09-12T06:50:06.150 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_41.status 2024-09-12T06:50:06.196 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 1 2024-09-12T06:50:06.196 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 1 2024-09-12T06:50:06.196 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:50:06.197 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=3630], [object_number=7, tag_tid=1, entry_tid=3627]]], state=connected] 2024-09-12T06:50:06.197 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=3630], [object_number=7, tag_tid=1, entry_tid=3627]]], state=connected] 2024-09-12T06:50:06.199 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_41.status 2024-09-12T06:50:06.201 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-12T06:50:06.201 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-09-12T06:50:06.202 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:50:06.202 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_snap_present cluster1 mirror image_41 snap 2024-09-12T06:50:06.202 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:50:06.202 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:06.202 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_41 2024-09-12T06:50:06.202 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:50:06.202 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:50:06.202 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T06:50:06.202 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:50:07.202 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 info mirror/image_41@snap 2024-09-12T06:50:07.249 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:rbd image 'image_41': 2024-09-12T06:50:07.249 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: size 128 MiB in 32 objects 2024-09-12T06:50:07.249 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: order 22 (4 MiB objects) 2024-09-12T06:50:07.249 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: snapshot_count: 1 2024-09-12T06:50:07.249 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: id: 10a89c5ff82b 2024-09-12T06:50:07.249 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: block_name_prefix: rbd_data.10a89c5ff82b 2024-09-12T06:50:07.249 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: format: 2 2024-09-12T06:50:07.249 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: features: layering, exclusive-lock, journaling 2024-09-12T06:50:07.249 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: op_features: 2024-09-12T06:50:07.249 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: flags: 2024-09-12T06:50:07.249 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: create_timestamp: Thu Sep 12 06:44:10 2024 2024-09-12T06:50:07.249 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: access_timestamp: Thu Sep 12 06:44:10 2024 2024-09-12T06:50:07.250 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: modify_timestamp: Thu Sep 12 06:45:11 2024 2024-09-12T06:50:07.250 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: protected: False 2024-09-12T06:50:07.250 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: journal: 10a89c5ff82b 2024-09-12T06:50:07.250 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring state: enabled 2024-09-12T06:50:07.250 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring mode: journal 2024-09-12T06:50:07.250 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring global id: e48558de-0577-4d46-a234-dfc9c26f24f0 2024-09-12T06:50:07.250 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring primary: false 2024-09-12T06:50:07.253 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:50:07.253 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror image_41 snap 2024-09-12T06:50:07.253 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:07.253 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_41 2024-09-12T06:50:07.253 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:50:07.253 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:50:07.253 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_41.export 2024-09-12T06:50:07.253 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_41.export 2024-09-12T06:50:07.253 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_41.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_41.export 2024-09-12T06:50:07.253 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export image_41@snap /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_41.export 2024-09-12T06:50:07.389 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:50:07.395 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export image_41@snap /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_41.export 2024-09-12T06:50:07.556 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:50:07.562 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_41.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_41.export 2024-09-12T06:50:07.617 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_41.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_41.export 2024-09-12T06:50:07.649 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:50:07.649 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:50:07.650 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_42 2024-09-12T06:50:07.650 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_snap cluster2 mirror image_42 snap 2024-09-12T06:50:07.650 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:50:07.650 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:07.650 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_42 2024-09-12T06:50:07.650 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:50:07.650 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror snap create image_42@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T06:50:09.827 INFO:tasks.ceph.cluster2.osd.2.smithi152.stderr:2024-09-12T06:50:09.828+0000 7f4789beb640 -1 reset not still connected to 0x56163d519e10 2024-09-12T06:50:09.828 INFO:tasks.ceph.cluster2.osd.2.smithi152.stderr:2024-09-12T06:50:09.828+0000 7f4789beb640 -1 reset not still connected to 0x56163df360d0 2024-09-12T06:50:09.828 INFO:tasks.ceph.cluster2.osd.2.smithi152.stderr:2024-09-12T06:50:09.828+0000 7f4789beb640 -1 reset not still connected to 0x56163df36270 2024-09-12T06:50:10.653 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_started cluster1 mirror image_42 2024-09-12T06:50:10.653 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:50:10.654 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:10.654 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_42 2024-09-12T06:50:10.654 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_state cluster1 mirror image_42 started 2024-09-12T06:50:10.654 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:50:10.654 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:10.654 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_42 2024-09-12T06:50:10.654 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=started 2024-09-12T06:50:10.654 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:50:10.654 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:50:10.654 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:50:11.655 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_replay_state cluster1 mirror image_42 started 2024-09-12T06:50:11.655 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:50:11.655 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:11.655 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_42 2024-09-12T06:50:11.655 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=started 2024-09-12T06:50:11.655 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local status_result 2024-09-12T06:50:11.655 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=stopped 2024-09-12T06:50:11.655 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_42 2024-09-12T06:50:11.656 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -i state 2024-09-12T06:50:11.656 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_instance=cluster1 2024-09-12T06:50:11.656 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:50:11.656 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:50:11.656 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance=cluster1 2024-09-12T06:50:11.656 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local loop_instance 2024-09-12T06:50:11.656 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:50:11.656 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sleep 0 2024-09-12T06:50:11.659 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:50:11.659 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ seq 0 1 2024-09-12T06:50:11.660 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:50:11.660 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_42 2024-09-12T06:50:11.660 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:0 2024-09-12T06:50:11.660 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:50:11.660 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:50:11.660 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:50:11.660 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:50:11.660 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:50:11.660 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:50:11.660 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:50:11.660 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:50:11.660 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:50:11.661 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:50:11.661 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:50:11.661 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:50:11.661 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:50:11.661 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:50:11.661 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:0 2024-09-12T06:50:11.661 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:50:11.661 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:50:11.661 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:50:11.661 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:0 2024-09-12T06:50:11.661 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:50:11.661 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:50:11.661 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:50:11.661 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:50:11.662 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T06:50:11.662 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:50:11.662 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:50:11.662 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=0 2024-09-12T06:50:11.662 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=0 2024-09-12T06:50:11.662 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:50:11.677 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:50:11.677 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:50:11.677 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:50:11.677 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_42 2024-09-12T06:50:11.771 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:Invalid command: unused arguments: ['mirror/image_42'] 2024-09-12T06:50:11.771 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T06:50:11.771 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:admin_socket: invalid command 2024-09-12T06:50:11.775 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:50:11.775 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_42 2024-09-12T06:50:11.775 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:1 2024-09-12T06:50:11.775 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:50:11.775 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:50:11.775 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T06:50:11.775 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:1 2024-09-12T06:50:11.776 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:50:11.776 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:50:11.776 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:50:11.776 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:50:11.776 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T06:50:11.776 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:50:11.776 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:50:11.776 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=1 2024-09-12T06:50:11.776 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:50:11.776 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T06:50:11.776 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:1 2024-09-12T06:50:11.776 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:50:11.776 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:50:11.777 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T06:50:11.777 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:1 2024-09-12T06:50:11.777 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:50:11.777 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:50:11.777 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:50:11.777 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:50:11.777 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T06:50:11.777 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:50:11.777 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:50:11.777 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=1 2024-09-12T06:50:11.777 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=1 2024-09-12T06:50:11.777 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T06:50:11.794 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:50:11.795 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:50:11.795 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:50:11.795 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_42 2024-09-12T06:50:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ return 0 2024-09-12T06:50:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T06:50:11.894 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo ' "state": "Replaying"' 2024-09-12T06:50:11.894 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep -i Replaying 2024-09-12T06:50:11.894 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: "state": "Replaying" 2024-09-12T06:50:11.896 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_state=started 2024-09-12T06:50:11.897 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test started = started 2024-09-12T06:50:11.897 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:50:11.897 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_42 2024-09-12T06:50:11.897 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:50:11.897 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:50:11.897 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:50:11.897 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:50:11.897 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_42 2024-09-12T06:50:11.897 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' journal = journal ']' 2024-09-12T06:50:11.897 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror image_42 2024-09-12T06:50:11.897 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:50:11.897 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:50:11.897 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:50:11.898 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:50:11.898 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_42 2024-09-12T06:50:11.898 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-12T06:50:11.898 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-12T06:50:11.898 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:50:11.898 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:50:11.898 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:50:12.097 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror image_42 2024-09-12T06:50:12.097 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:50:12.097 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:12.097 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_42 2024-09-12T06:50:12.097 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:50:12.097 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n image_42 ']' 2024-09-12T06:50:12.097 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/image_42' 2024-09-12T06:50:12.097 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_42 2024-09-12T06:50:12.097 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:50:12.098 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:50:12.098 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:50:12.098 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:50:12.098 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:50:12.098 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:50:12.098 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:50:12.098 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:50:12.098 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:50:12.102 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:50:12.102 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_42 2024-09-12T06:50:12.102 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:50:12.102 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:50:12.102 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:50:12.102 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:50:12.102 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:50:12.102 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:50:12.103 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:50:12.103 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:50:12.103 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:50:12.103 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:50:12.103 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:50:12.103 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:50:12.103 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:50:12.103 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:50:12.103 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:50:12.103 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:50:12.103 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:50:12.103 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:50:12.103 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:50:12.104 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:50:12.104 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:50:12.104 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:50:12.104 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:50:12.104 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:50:12.104 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:50:12.104 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:50:12.104 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:50:12.104 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:50:12.105 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:50:12.105 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:50:12.118 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:50:12.118 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:50:12.118 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:50:12.118 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_42 2024-09-12T06:50:12.212 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:Invalid command: unused arguments: ['mirror/image_42'] 2024-09-12T06:50:12.212 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd mirror flush : flush rbd mirror 2024-09-12T06:50:12.212 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:admin_socket: invalid command 2024-09-12T06:50:12.212 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:50:12.212 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_42 2024-09-12T06:50:12.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:1 2024-09-12T06:50:12.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:50:12.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:50:12.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-12T06:50:12.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:1 2024-09-12T06:50:12.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:50:12.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:50:12.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:50:12.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=1 2024-09-12T06:50:12.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-12T06:50:12.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:50:12.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:50:12.214 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=1 2024-09-12T06:50:12.214 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:50:12.214 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-12T06:50:12.214 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:1 2024-09-12T06:50:12.214 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:50:12.214 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:50:12.214 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T06:50:12.214 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:1 2024-09-12T06:50:12.214 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:50:12.214 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:50:12.214 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:50:12.214 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:50:12.215 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T06:50:12.215 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:50:12.215 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:50:12.215 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=1 2024-09-12T06:50:12.215 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:50:12.215 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T06:50:12.227 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:50:12.227 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:50:12.227 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:50:12.227 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_42 2024-09-12T06:50:12.329 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:50:12.329 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror image_42 2024-09-12T06:50:12.329 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:50:12.329 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:50:12.329 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_42 2024-09-12T06:50:12.330 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_42 '' 2024-09-12T06:50:12.330 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:50:12.330 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:50:12.330 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_42 2024-09-12T06:50:12.330 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:50:12.330 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_42.status 2024-09-12T06:50:12.330 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_42.status 2024-09-12T06:50:12.330 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:50:12.332 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_42.status 2024-09-12T06:50:12.332 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_42 2024-09-12T06:50:12.332 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_42.status 2024-09-12T06:50:12.374 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 2 2024-09-12T06:50:12.374 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 2 2024-09-12T06:50:12.374 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:50:12.374 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8255], [object_number=10, tag_tid=1, entry_tid=8254]]], state=connected] 2024-09-12T06:50:12.374 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8255], [object_number=10, tag_tid=1, entry_tid=8254]]], state=connected] 2024-09-12T06:50:12.377 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_42.status 2024-09-12T06:50:12.378 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-09-12T06:50:12.379 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror image_42 2024-09-12T06:50:12.379 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:50:12.379 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:50:12.379 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_42 2024-09-12T06:50:12.379 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_42 '..*' 2024-09-12T06:50:12.379 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:50:12.380 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:50:12.380 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_42 2024-09-12T06:50:12.380 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:50:12.380 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_42.status 2024-09-12T06:50:12.380 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_42.status 2024-09-12T06:50:12.380 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:50:12.380 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_42.status 2024-09-12T06:50:12.381 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_42 2024-09-12T06:50:12.381 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_42.status 2024-09-12T06:50:12.426 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 2 2024-09-12T06:50:12.426 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 2 2024-09-12T06:50:12.426 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:50:12.426 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8255], [object_number=10, tag_tid=1, entry_tid=8254]]], state=connected] 2024-09-12T06:50:12.426 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8255], [object_number=10, tag_tid=1, entry_tid=8254]]], state=connected] 2024-09-12T06:50:12.428 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_42.status 2024-09-12T06:50:12.431 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-09-12T06:50:12.431 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-09-12T06:50:12.431 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:50:12.431 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_snap_present cluster1 mirror image_42 snap 2024-09-12T06:50:12.431 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:50:12.431 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:12.431 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_42 2024-09-12T06:50:12.431 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:50:12.431 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:50:12.431 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T06:50:12.431 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:50:13.431 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 info mirror/image_42@snap 2024-09-12T06:50:13.479 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:rbd image 'image_42': 2024-09-12T06:50:13.479 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: size 128 MiB in 32 objects 2024-09-12T06:50:13.480 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: order 22 (4 MiB objects) 2024-09-12T06:50:13.480 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: snapshot_count: 1 2024-09-12T06:50:13.480 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: id: 10af2e928be9 2024-09-12T06:50:13.480 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: block_name_prefix: rbd_data.10af2e928be9 2024-09-12T06:50:13.480 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: format: 2 2024-09-12T06:50:13.480 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: features: layering, exclusive-lock, journaling 2024-09-12T06:50:13.480 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: op_features: 2024-09-12T06:50:13.480 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: flags: 2024-09-12T06:50:13.480 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: create_timestamp: Thu Sep 12 06:44:15 2024 2024-09-12T06:50:13.480 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: access_timestamp: Thu Sep 12 06:44:15 2024 2024-09-12T06:50:13.480 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: modify_timestamp: Thu Sep 12 06:45:16 2024 2024-09-12T06:50:13.480 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: protected: False 2024-09-12T06:50:13.480 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: journal: 10af2e928be9 2024-09-12T06:50:13.481 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring state: enabled 2024-09-12T06:50:13.481 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring mode: journal 2024-09-12T06:50:13.481 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring global id: 92204849-0801-49e6-b243-476b5ce5d37c 2024-09-12T06:50:13.481 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring primary: false 2024-09-12T06:50:13.483 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:50:13.483 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror image_42 snap 2024-09-12T06:50:13.483 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:13.483 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_42 2024-09-12T06:50:13.483 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:50:13.484 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:50:13.484 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_42.export 2024-09-12T06:50:13.484 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_42.export 2024-09-12T06:50:13.484 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_42.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_42.export 2024-09-12T06:50:13.484 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export image_42@snap /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_42.export 2024-09-12T06:50:13.635 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:50:13.641 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export image_42@snap /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_42.export 2024-09-12T06:50:13.866 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:50:13.872 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_42.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_42.export 2024-09-12T06:50:13.927 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_42.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_42.export 2024-09-12T06:50:13.961 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:50:13.961 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:50:13.961 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_43 2024-09-12T06:50:13.961 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_snap cluster2 mirror image_43 snap 2024-09-12T06:50:13.961 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:50:13.962 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:13.962 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_43 2024-09-12T06:50:13.962 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:50:13.962 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror snap create image_43@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T06:50:16.666 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_started cluster1 mirror image_43 2024-09-12T06:50:16.667 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:50:16.667 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:16.667 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_43 2024-09-12T06:50:16.667 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_state cluster1 mirror image_43 started 2024-09-12T06:50:16.667 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:50:16.667 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:16.667 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_43 2024-09-12T06:50:16.667 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=started 2024-09-12T06:50:16.667 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:50:16.667 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:50:16.667 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:50:17.668 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_replay_state cluster1 mirror image_43 started 2024-09-12T06:50:17.669 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:50:17.669 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:17.669 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_43 2024-09-12T06:50:17.669 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=started 2024-09-12T06:50:17.669 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local status_result 2024-09-12T06:50:17.669 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=stopped 2024-09-12T06:50:17.669 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_43 2024-09-12T06:50:17.669 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_instance=cluster1 2024-09-12T06:50:17.669 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -i state 2024-09-12T06:50:17.669 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:50:17.669 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:50:17.669 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance=cluster1 2024-09-12T06:50:17.670 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local loop_instance 2024-09-12T06:50:17.670 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:50:17.670 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sleep 0 2024-09-12T06:50:17.673 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:50:17.673 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ seq 0 1 2024-09-12T06:50:17.673 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:50:17.673 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_43 2024-09-12T06:50:17.673 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:0 2024-09-12T06:50:17.674 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:50:17.674 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:50:17.674 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:50:17.674 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:50:17.674 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:50:17.674 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:50:17.674 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:50:17.674 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:50:17.674 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:50:17.674 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:50:17.674 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:50:17.674 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:50:17.675 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:50:17.675 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:50:17.675 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:0 2024-09-12T06:50:17.675 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:50:17.675 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:50:17.675 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:50:17.675 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:0 2024-09-12T06:50:17.675 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:50:17.675 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:50:17.675 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:50:17.675 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:50:17.675 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T06:50:17.676 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:50:17.676 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:50:17.676 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=0 2024-09-12T06:50:17.676 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=0 2024-09-12T06:50:17.676 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:50:17.693 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:50:17.693 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:50:17.693 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:50:17.693 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_43 2024-09-12T06:50:17.794 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ return 0 2024-09-12T06:50:17.795 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T06:50:17.795 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo ' "state": "Replaying"' 2024-09-12T06:50:17.795 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep -i Replaying 2024-09-12T06:50:17.796 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: "state": "Replaying" 2024-09-12T06:50:17.798 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_state=started 2024-09-12T06:50:17.798 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test started = started 2024-09-12T06:50:17.798 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:50:17.798 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_43 2024-09-12T06:50:17.798 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:50:17.798 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:50:17.798 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:50:17.798 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:50:17.798 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_43 2024-09-12T06:50:17.798 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' journal = journal ']' 2024-09-12T06:50:17.798 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror image_43 2024-09-12T06:50:17.799 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:50:17.799 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:50:17.799 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:50:17.799 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:50:17.799 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_43 2024-09-12T06:50:17.799 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-12T06:50:17.799 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-12T06:50:17.799 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:50:17.799 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:50:17.799 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:50:17.999 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror image_43 2024-09-12T06:50:17.999 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:50:17.999 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:17.999 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_43 2024-09-12T06:50:17.999 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:50:17.999 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n image_43 ']' 2024-09-12T06:50:17.999 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/image_43' 2024-09-12T06:50:17.999 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_43 2024-09-12T06:50:17.999 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:50:18.000 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:50:18.000 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:50:18.000 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:50:18.000 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:50:18.000 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:50:18.000 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:50:18.000 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:50:18.000 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:50:18.003 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:50:18.003 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_43 2024-09-12T06:50:18.003 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:50:18.003 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:50:18.004 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:50:18.004 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:50:18.004 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:50:18.004 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:50:18.004 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:50:18.004 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:50:18.004 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:50:18.004 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:50:18.004 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:50:18.004 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:50:18.004 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:50:18.005 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:50:18.005 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:50:18.005 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:50:18.005 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:50:18.005 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:50:18.005 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:50:18.005 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:50:18.005 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:50:18.005 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:50:18.005 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:50:18.005 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:50:18.005 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:50:18.005 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:50:18.006 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:50:18.006 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:50:18.006 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:50:18.006 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:50:18.020 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:50:18.020 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:50:18.020 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:50:18.020 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_43 2024-09-12T06:50:18.118 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:50:18.119 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror image_43 2024-09-12T06:50:18.119 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:50:18.119 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:50:18.119 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_43 2024-09-12T06:50:18.119 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_43 '' 2024-09-12T06:50:18.119 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:50:18.119 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:50:18.119 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_43 2024-09-12T06:50:18.119 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:50:18.119 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_43.status 2024-09-12T06:50:18.119 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_43.status 2024-09-12T06:50:18.119 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:50:18.120 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_43.status 2024-09-12T06:50:18.120 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_43 2024-09-12T06:50:18.120 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_43.status 2024-09-12T06:50:18.167 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 2 2024-09-12T06:50:18.167 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 2 2024-09-12T06:50:18.167 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:50:18.167 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=8546], [object_number=11, tag_tid=1, entry_tid=8543]]], state=connected] 2024-09-12T06:50:18.167 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=8546], [object_number=11, tag_tid=1, entry_tid=8543]]], state=connected] 2024-09-12T06:50:18.170 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_43.status 2024-09-12T06:50:18.171 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-09-12T06:50:18.173 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror image_43 2024-09-12T06:50:18.174 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:50:18.174 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:50:18.174 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_43 2024-09-12T06:50:18.174 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_43 '..*' 2024-09-12T06:50:18.174 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:50:18.174 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:50:18.174 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_43 2024-09-12T06:50:18.174 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:50:18.174 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_43.status 2024-09-12T06:50:18.175 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_43.status 2024-09-12T06:50:18.175 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:50:18.175 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_43.status 2024-09-12T06:50:18.175 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_43 2024-09-12T06:50:18.175 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_43.status 2024-09-12T06:50:18.222 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 2 2024-09-12T06:50:18.222 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 2 2024-09-12T06:50:18.222 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:50:18.222 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=8546], [object_number=11, tag_tid=1, entry_tid=8543]]], state=connected] 2024-09-12T06:50:18.222 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=8546], [object_number=11, tag_tid=1, entry_tid=8543]]], state=connected] 2024-09-12T06:50:18.225 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_43.status 2024-09-12T06:50:18.227 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-09-12T06:50:18.228 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-09-12T06:50:18.228 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:50:18.228 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_snap_present cluster1 mirror image_43 snap 2024-09-12T06:50:18.228 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:50:18.228 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:18.228 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_43 2024-09-12T06:50:18.228 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:50:18.228 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:50:18.228 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T06:50:18.228 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:50:19.228 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 info mirror/image_43@snap 2024-09-12T06:50:19.273 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:rbd image 'image_43': 2024-09-12T06:50:19.273 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: size 128 MiB in 32 objects 2024-09-12T06:50:19.273 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: order 22 (4 MiB objects) 2024-09-12T06:50:19.274 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: snapshot_count: 1 2024-09-12T06:50:19.274 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: id: 10a85792e1a1 2024-09-12T06:50:19.274 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: block_name_prefix: rbd_data.10a85792e1a1 2024-09-12T06:50:19.274 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: format: 2 2024-09-12T06:50:19.274 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: features: layering, exclusive-lock, journaling 2024-09-12T06:50:19.274 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: op_features: 2024-09-12T06:50:19.274 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: flags: 2024-09-12T06:50:19.274 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: create_timestamp: Thu Sep 12 06:44:24 2024 2024-09-12T06:50:19.274 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: access_timestamp: Thu Sep 12 06:44:24 2024 2024-09-12T06:50:19.274 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: modify_timestamp: Thu Sep 12 06:45:25 2024 2024-09-12T06:50:19.274 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: protected: False 2024-09-12T06:50:19.274 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: journal: 10a85792e1a1 2024-09-12T06:50:19.274 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring state: enabled 2024-09-12T06:50:19.275 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring mode: journal 2024-09-12T06:50:19.275 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring global id: 58e5e346-d4f7-47c0-b30e-9d2d132daf26 2024-09-12T06:50:19.275 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring primary: false 2024-09-12T06:50:19.277 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:50:19.277 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror image_43 snap 2024-09-12T06:50:19.277 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:19.277 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_43 2024-09-12T06:50:19.277 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:50:19.277 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:50:19.277 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_43.export 2024-09-12T06:50:19.278 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_43.export 2024-09-12T06:50:19.278 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_43.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_43.export 2024-09-12T06:50:19.278 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export image_43@snap /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_43.export 2024-09-12T06:50:19.431 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:50:19.437 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export image_43@snap /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_43.export 2024-09-12T06:50:19.649 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:50:19.655 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_43.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_43.export 2024-09-12T06:50:19.710 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_43.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_43.export 2024-09-12T06:50:19.743 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:50:19.743 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:50:19.743 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_44 2024-09-12T06:50:19.743 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_snap cluster2 mirror image_44 snap 2024-09-12T06:50:19.743 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:50:19.743 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:19.743 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_44 2024-09-12T06:50:19.743 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:50:19.744 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror snap create image_44@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T06:50:21.690 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_started cluster1 mirror image_44 2024-09-12T06:50:21.691 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:50:21.691 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:21.691 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_44 2024-09-12T06:50:21.691 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_state cluster1 mirror image_44 started 2024-09-12T06:50:21.691 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:50:21.691 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:21.691 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_44 2024-09-12T06:50:21.691 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=started 2024-09-12T06:50:21.691 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:50:21.691 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:50:21.691 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:50:22.692 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_replay_state cluster1 mirror image_44 started 2024-09-12T06:50:22.692 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:50:22.693 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:22.693 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_44 2024-09-12T06:50:22.693 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=started 2024-09-12T06:50:22.693 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local status_result 2024-09-12T06:50:22.693 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=stopped 2024-09-12T06:50:22.693 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_44 2024-09-12T06:50:22.693 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -i state 2024-09-12T06:50:22.693 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_instance=cluster1 2024-09-12T06:50:22.693 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:50:22.693 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:50:22.693 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance=cluster1 2024-09-12T06:50:22.693 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local loop_instance 2024-09-12T06:50:22.693 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:50:22.694 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sleep 0 2024-09-12T06:50:22.694 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:50:22.696 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ seq 0 1 2024-09-12T06:50:22.696 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:50:22.696 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_44 2024-09-12T06:50:22.696 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:0 2024-09-12T06:50:22.696 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:50:22.696 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:50:22.696 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:50:22.696 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:50:22.696 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:50:22.697 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:50:22.697 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:50:22.697 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:50:22.697 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:50:22.697 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:50:22.697 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:50:22.697 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:50:22.697 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:50:22.697 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:50:22.697 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:0 2024-09-12T06:50:22.697 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:50:22.697 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:50:22.697 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:50:22.699 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:0 2024-09-12T06:50:22.699 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:50:22.699 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:50:22.699 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:50:22.699 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:50:22.699 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T06:50:22.699 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:50:22.700 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:50:22.700 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=0 2024-09-12T06:50:22.700 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=0 2024-09-12T06:50:22.700 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:50:22.714 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:50:22.714 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:50:22.714 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:50:22.714 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_44 2024-09-12T06:50:22.806 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2024-09-12T06:50:22.806 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T06:50:22.806 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:admin_socket: invalid command 2024-09-12T06:50:22.810 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:50:22.810 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_44 2024-09-12T06:50:22.810 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:1 2024-09-12T06:50:22.810 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:50:22.810 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:50:22.810 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T06:50:22.810 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:1 2024-09-12T06:50:22.810 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:50:22.810 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:50:22.810 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:50:22.810 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:50:22.810 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T06:50:22.811 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:50:22.811 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:50:22.811 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=1 2024-09-12T06:50:22.811 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:50:22.811 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T06:50:22.812 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:1 2024-09-12T06:50:22.812 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:50:22.812 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:50:22.812 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T06:50:22.812 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:1 2024-09-12T06:50:22.812 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:50:22.812 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:50:22.812 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:50:22.812 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:50:22.812 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T06:50:22.812 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:50:22.813 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:50:22.813 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=1 2024-09-12T06:50:22.813 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=1 2024-09-12T06:50:22.813 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T06:50:22.828 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:50:22.828 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:50:22.828 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:50:22.828 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_44 2024-09-12T06:50:22.927 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ return 0 2024-09-12T06:50:22.927 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T06:50:22.928 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo ' "state": "Replaying"' 2024-09-12T06:50:22.928 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep -i Replaying 2024-09-12T06:50:22.928 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: "state": "Replaying" 2024-09-12T06:50:22.930 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_state=started 2024-09-12T06:50:22.930 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test started = started 2024-09-12T06:50:22.930 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:50:22.930 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_44 2024-09-12T06:50:22.930 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:50:22.930 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:50:22.930 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:50:22.930 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:50:22.930 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_44 2024-09-12T06:50:22.931 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' journal = journal ']' 2024-09-12T06:50:22.931 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror image_44 2024-09-12T06:50:22.931 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:50:22.931 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:50:22.931 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:50:22.931 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:50:22.931 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_44 2024-09-12T06:50:22.931 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-12T06:50:22.931 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-12T06:50:22.931 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:50:22.931 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:50:22.931 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:50:23.131 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror image_44 2024-09-12T06:50:23.131 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:50:23.131 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:23.131 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_44 2024-09-12T06:50:23.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:50:23.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n image_44 ']' 2024-09-12T06:50:23.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/image_44' 2024-09-12T06:50:23.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_44 2024-09-12T06:50:23.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:50:23.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:50:23.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:50:23.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:50:23.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:50:23.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:50:23.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:50:23.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:50:23.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:50:23.135 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:50:23.136 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_44 2024-09-12T06:50:23.136 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:50:23.136 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:50:23.136 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:50:23.136 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:50:23.136 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:50:23.136 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:50:23.136 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:50:23.136 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:50:23.136 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:50:23.136 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:50:23.136 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:50:23.136 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:50:23.137 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:50:23.137 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:50:23.137 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:50:23.137 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:50:23.137 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:50:23.137 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:50:23.137 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:50:23.137 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:50:23.137 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:50:23.137 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:50:23.137 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:50:23.137 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:50:23.137 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:50:23.138 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:50:23.138 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:50:23.138 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:50:23.138 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:50:23.138 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:50:23.154 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:50:23.154 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:50:23.154 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:50:23.154 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_44 2024-09-12T06:50:23.242 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2024-09-12T06:50:23.243 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd mirror flush : flush rbd mirror 2024-09-12T06:50:23.243 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:admin_socket: invalid command 2024-09-12T06:50:23.246 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:50:23.246 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_44 2024-09-12T06:50:23.247 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:1 2024-09-12T06:50:23.247 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:50:23.247 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:50:23.247 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-12T06:50:23.247 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:1 2024-09-12T06:50:23.247 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:50:23.247 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:50:23.247 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:50:23.247 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=1 2024-09-12T06:50:23.247 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-12T06:50:23.247 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:50:23.247 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:50:23.247 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=1 2024-09-12T06:50:23.248 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:50:23.248 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-12T06:50:23.248 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:1 2024-09-12T06:50:23.248 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:50:23.248 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:50:23.248 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T06:50:23.248 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:1 2024-09-12T06:50:23.248 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:50:23.248 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:50:23.248 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:50:23.248 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:50:23.248 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T06:50:23.248 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:50:23.249 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:50:23.249 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=1 2024-09-12T06:50:23.249 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:50:23.249 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T06:50:23.266 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:50:23.266 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:50:23.266 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:50:23.266 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_44 2024-09-12T06:50:23.365 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:50:23.365 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror image_44 2024-09-12T06:50:23.365 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:50:23.365 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:50:23.365 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_44 2024-09-12T06:50:23.365 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_44 '' 2024-09-12T06:50:23.365 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:50:23.365 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:50:23.365 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_44 2024-09-12T06:50:23.365 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:50:23.365 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_44.status 2024-09-12T06:50:23.366 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_44.status 2024-09-12T06:50:23.366 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:50:23.366 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_44.status 2024-09-12T06:50:23.366 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_44 2024-09-12T06:50:23.366 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_44.status 2024-09-12T06:50:23.411 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 2 2024-09-12T06:50:23.412 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 3 2024-09-12T06:50:23.412 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:50:23.412 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=10246], [object_number=11, tag_tid=1, entry_tid=10243]]], state=connected] 2024-09-12T06:50:23.412 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=10246], [object_number=11, tag_tid=1, entry_tid=10243]]], state=connected] 2024-09-12T06:50:23.414 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_44.status 2024-09-12T06:50:23.416 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-09-12T06:50:23.417 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror image_44 2024-09-12T06:50:23.417 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:50:23.417 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:50:23.417 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_44 2024-09-12T06:50:23.417 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_44 '..*' 2024-09-12T06:50:23.418 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:50:23.418 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:50:23.418 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_44 2024-09-12T06:50:23.418 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:50:23.418 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_44.status 2024-09-12T06:50:23.418 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_44.status 2024-09-12T06:50:23.418 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:50:23.418 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_44.status 2024-09-12T06:50:23.419 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_44 2024-09-12T06:50:23.419 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_44.status 2024-09-12T06:50:23.464 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 2 2024-09-12T06:50:23.464 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 3 2024-09-12T06:50:23.464 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:50:23.464 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=10246], [object_number=11, tag_tid=1, entry_tid=10243]]], state=connected] 2024-09-12T06:50:23.464 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=10246], [object_number=11, tag_tid=1, entry_tid=10243]]], state=connected] 2024-09-12T06:50:23.467 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_44.status 2024-09-12T06:50:23.470 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-09-12T06:50:23.470 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=1' 2024-09-12T06:50:23.470 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:50:23.470 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_snap_present cluster1 mirror image_44 snap 2024-09-12T06:50:23.470 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:50:23.471 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:23.471 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_44 2024-09-12T06:50:23.471 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:50:23.471 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:50:23.471 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T06:50:23.471 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:50:24.470 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 info mirror/image_44@snap 2024-09-12T06:50:24.518 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:rbd image 'image_44': 2024-09-12T06:50:24.518 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: size 128 MiB in 32 objects 2024-09-12T06:50:24.518 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: order 22 (4 MiB objects) 2024-09-12T06:50:24.518 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: snapshot_count: 1 2024-09-12T06:50:24.518 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: id: 10af294efce1 2024-09-12T06:50:24.518 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: block_name_prefix: rbd_data.10af294efce1 2024-09-12T06:50:24.518 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: format: 2 2024-09-12T06:50:24.519 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: features: layering, exclusive-lock, journaling 2024-09-12T06:50:24.519 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: op_features: 2024-09-12T06:50:24.519 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: flags: 2024-09-12T06:50:24.519 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: create_timestamp: Thu Sep 12 06:44:35 2024 2024-09-12T06:50:24.519 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: access_timestamp: Thu Sep 12 06:44:35 2024 2024-09-12T06:50:24.519 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: modify_timestamp: Thu Sep 12 06:45:36 2024 2024-09-12T06:50:24.519 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: protected: False 2024-09-12T06:50:24.519 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: journal: 10af294efce1 2024-09-12T06:50:24.519 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring state: enabled 2024-09-12T06:50:24.519 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring mode: journal 2024-09-12T06:50:24.519 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring global id: 47924219-df33-491d-b285-4413bfeff82a 2024-09-12T06:50:24.519 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring primary: false 2024-09-12T06:50:24.522 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:50:24.522 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror image_44 snap 2024-09-12T06:50:24.522 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:24.522 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_44 2024-09-12T06:50:24.522 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:50:24.522 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:50:24.522 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_44.export 2024-09-12T06:50:24.522 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_44.export 2024-09-12T06:50:24.522 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_44.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_44.export 2024-09-12T06:50:24.523 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export image_44@snap /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_44.export 2024-09-12T06:50:24.614 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:50:24.620 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export image_44@snap /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_44.export 2024-09-12T06:50:24.820 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:50:24.830 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_44.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_44.export 2024-09-12T06:50:24.923 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_44.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_44.export 2024-09-12T06:50:24.940 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:50:24.941 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:50:24.941 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_45 2024-09-12T06:50:24.941 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_snap cluster2 mirror image_45 snap 2024-09-12T06:50:24.941 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:50:24.941 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:24.941 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_45 2024-09-12T06:50:24.941 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:50:24.941 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror snap create image_45@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T06:50:27.696 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_started cluster1 mirror image_45 2024-09-12T06:50:27.696 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:50:27.696 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:27.696 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_45 2024-09-12T06:50:27.696 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_state cluster1 mirror image_45 started 2024-09-12T06:50:27.697 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:50:27.697 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:27.697 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_45 2024-09-12T06:50:27.697 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=started 2024-09-12T06:50:27.697 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:50:27.697 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:50:27.697 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:50:28.698 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_replay_state cluster1 mirror image_45 started 2024-09-12T06:50:28.698 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:50:28.698 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:28.698 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_45 2024-09-12T06:50:28.698 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=started 2024-09-12T06:50:28.698 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local status_result 2024-09-12T06:50:28.699 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=stopped 2024-09-12T06:50:28.699 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_45 2024-09-12T06:50:28.699 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_instance=cluster1 2024-09-12T06:50:28.699 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -i state 2024-09-12T06:50:28.699 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:50:28.699 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:50:28.699 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance=cluster1 2024-09-12T06:50:28.699 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local loop_instance 2024-09-12T06:50:28.699 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:50:28.699 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sleep 0 2024-09-12T06:50:28.702 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:50:28.703 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ seq 0 1 2024-09-12T06:50:28.703 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:50:28.703 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_45 2024-09-12T06:50:28.703 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:0 2024-09-12T06:50:28.703 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:50:28.703 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:50:28.703 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:50:28.703 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:50:28.703 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:50:28.703 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:50:28.703 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:50:28.703 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:50:28.704 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:50:28.704 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:50:28.704 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:50:28.704 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:50:28.704 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:50:28.704 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:50:28.704 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:0 2024-09-12T06:50:28.704 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:50:28.704 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:50:28.704 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:50:28.704 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:0 2024-09-12T06:50:28.704 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:50:28.704 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:50:28.704 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:50:28.705 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:50:28.705 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T06:50:28.705 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:50:28.705 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:50:28.705 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=0 2024-09-12T06:50:28.705 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=0 2024-09-12T06:50:28.705 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:50:28.722 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:50:28.722 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:50:28.722 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:50:28.722 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_45 2024-09-12T06:50:28.821 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ return 0 2024-09-12T06:50:28.822 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T06:50:28.822 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo ' "state": "Replaying"' 2024-09-12T06:50:28.822 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep -i Replaying 2024-09-12T06:50:28.823 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: "state": "Replaying" 2024-09-12T06:50:28.825 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_state=started 2024-09-12T06:50:28.825 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test started = started 2024-09-12T06:50:28.825 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:50:28.825 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_45 2024-09-12T06:50:28.825 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:50:28.826 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:50:28.826 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:50:28.826 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:50:28.826 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_45 2024-09-12T06:50:28.826 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' journal = journal ']' 2024-09-12T06:50:28.826 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror image_45 2024-09-12T06:50:28.826 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:50:28.826 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:50:28.826 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:50:28.826 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:50:28.826 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_45 2024-09-12T06:50:28.826 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-12T06:50:28.827 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-12T06:50:28.827 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:50:28.827 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:50:28.827 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:50:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror image_45 2024-09-12T06:50:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:50:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_45 2024-09-12T06:50:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:50:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n image_45 ']' 2024-09-12T06:50:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/image_45' 2024-09-12T06:50:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_45 2024-09-12T06:50:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:50:29.027 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:50:29.027 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:50:29.027 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:50:29.027 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:50:29.027 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:50:29.027 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:50:29.027 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:50:29.027 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:50:29.030 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:50:29.030 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_45 2024-09-12T06:50:29.030 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:50:29.030 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:50:29.030 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:50:29.030 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:50:29.030 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:50:29.030 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:50:29.030 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:50:29.030 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:50:29.030 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:50:29.030 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:50:29.030 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:50:29.031 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:50:29.031 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:50:29.031 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:50:29.031 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:50:29.031 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:50:29.031 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:50:29.031 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:50:29.031 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:50:29.031 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:50:29.031 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:50:29.031 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:50:29.031 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:50:29.031 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:50:29.031 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:50:29.032 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:50:29.032 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:50:29.032 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:50:29.032 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:50:29.032 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:50:29.047 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:50:29.047 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:50:29.047 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:50:29.047 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_45 2024-09-12T06:50:29.145 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:50:29.145 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror image_45 2024-09-12T06:50:29.145 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:50:29.145 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:50:29.145 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_45 2024-09-12T06:50:29.146 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_45 '' 2024-09-12T06:50:29.146 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:50:29.146 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:50:29.146 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_45 2024-09-12T06:50:29.146 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:50:29.146 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_45.status 2024-09-12T06:50:29.146 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_45.status 2024-09-12T06:50:29.146 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:50:29.147 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_45.status 2024-09-12T06:50:29.147 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_45 2024-09-12T06:50:29.147 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_45.status 2024-09-12T06:50:29.192 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 2 2024-09-12T06:50:29.192 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 2 2024-09-12T06:50:29.192 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:50:29.193 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8583], [object_number=10, tag_tid=1, entry_tid=8582]]], state=connected] 2024-09-12T06:50:29.193 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8583], [object_number=10, tag_tid=1, entry_tid=8582]]], state=connected] 2024-09-12T06:50:29.195 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_45.status 2024-09-12T06:50:29.197 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-09-12T06:50:29.197 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror image_45 2024-09-12T06:50:29.197 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:50:29.197 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:50:29.197 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_45 2024-09-12T06:50:29.197 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_45 '..*' 2024-09-12T06:50:29.198 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:50:29.198 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:50:29.198 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_45 2024-09-12T06:50:29.198 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:50:29.198 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_45.status 2024-09-12T06:50:29.198 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_45.status 2024-09-12T06:50:29.198 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:50:29.198 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_45.status 2024-09-12T06:50:29.199 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_45 2024-09-12T06:50:29.199 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_45.status 2024-09-12T06:50:29.245 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 2 2024-09-12T06:50:29.245 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 2 2024-09-12T06:50:29.245 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:50:29.245 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8583], [object_number=10, tag_tid=1, entry_tid=8582]]], state=connected] 2024-09-12T06:50:29.245 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8583], [object_number=10, tag_tid=1, entry_tid=8582]]], state=connected] 2024-09-12T06:50:29.248 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_45.status 2024-09-12T06:50:29.250 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-09-12T06:50:29.250 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-09-12T06:50:29.250 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:50:29.250 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_snap_present cluster1 mirror image_45 snap 2024-09-12T06:50:29.251 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:50:29.251 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:29.251 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_45 2024-09-12T06:50:29.251 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:50:29.251 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:50:29.251 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T06:50:29.251 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:50:30.251 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 info mirror/image_45@snap 2024-09-12T06:50:30.298 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:rbd image 'image_45': 2024-09-12T06:50:30.298 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: size 128 MiB in 32 objects 2024-09-12T06:50:30.298 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: order 22 (4 MiB objects) 2024-09-12T06:50:30.298 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: snapshot_count: 1 2024-09-12T06:50:30.298 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: id: 10a82009c454 2024-09-12T06:50:30.298 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: block_name_prefix: rbd_data.10a82009c454 2024-09-12T06:50:30.298 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: format: 2 2024-09-12T06:50:30.298 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: features: layering, exclusive-lock, journaling 2024-09-12T06:50:30.298 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: op_features: 2024-09-12T06:50:30.298 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: flags: 2024-09-12T06:50:30.299 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: create_timestamp: Thu Sep 12 06:44:42 2024 2024-09-12T06:50:30.299 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: access_timestamp: Thu Sep 12 06:44:42 2024 2024-09-12T06:50:30.299 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: modify_timestamp: Thu Sep 12 06:45:43 2024 2024-09-12T06:50:30.299 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: protected: False 2024-09-12T06:50:30.299 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: journal: 10a82009c454 2024-09-12T06:50:30.299 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring state: enabled 2024-09-12T06:50:30.299 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring mode: journal 2024-09-12T06:50:30.299 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring global id: 51155ae3-4aba-48d8-a442-a31245024a01 2024-09-12T06:50:30.299 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring primary: false 2024-09-12T06:50:30.302 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:50:30.302 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror image_45 snap 2024-09-12T06:50:30.302 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:30.302 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_45 2024-09-12T06:50:30.302 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:50:30.302 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:50:30.302 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_45.export 2024-09-12T06:50:30.303 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_45.export 2024-09-12T06:50:30.303 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_45.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_45.export 2024-09-12T06:50:30.303 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export image_45@snap /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_45.export 2024-09-12T06:50:30.451 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:50:30.458 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export image_45@snap /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_45.export 2024-09-12T06:50:30.647 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:50:30.659 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_45.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_45.export 2024-09-12T06:50:30.713 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_45.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_45.export 2024-09-12T06:50:30.745 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:50:30.745 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:50:30.745 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_46 2024-09-12T06:50:30.745 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_snap cluster2 mirror image_46 snap 2024-09-12T06:50:30.745 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:50:30.745 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:30.745 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_46 2024-09-12T06:50:30.746 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:50:30.746 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror snap create image_46@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T06:50:32.711 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_started cluster1 mirror image_46 2024-09-12T06:50:32.711 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:50:32.712 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:32.712 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_46 2024-09-12T06:50:32.712 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_state cluster1 mirror image_46 started 2024-09-12T06:50:32.712 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:50:32.712 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:32.712 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_46 2024-09-12T06:50:32.712 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=started 2024-09-12T06:50:32.712 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:50:32.712 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:50:32.712 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:50:33.713 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_replay_state cluster1 mirror image_46 started 2024-09-12T06:50:33.713 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:50:33.713 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:33.713 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_46 2024-09-12T06:50:33.713 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=started 2024-09-12T06:50:33.713 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local status_result 2024-09-12T06:50:33.713 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=stopped 2024-09-12T06:50:33.713 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_46 2024-09-12T06:50:33.714 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_instance=cluster1 2024-09-12T06:50:33.714 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -i state 2024-09-12T06:50:33.714 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:50:33.714 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:50:33.714 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance=cluster1 2024-09-12T06:50:33.714 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local loop_instance 2024-09-12T06:50:33.714 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:50:33.714 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sleep 0 2024-09-12T06:50:33.717 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:50:33.718 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ seq 0 1 2024-09-12T06:50:33.718 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:50:33.718 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_46 2024-09-12T06:50:33.718 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:0 2024-09-12T06:50:33.718 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:50:33.718 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:50:33.718 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:50:33.718 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:50:33.718 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:50:33.718 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:50:33.718 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:50:33.718 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:50:33.718 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:50:33.719 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:50:33.719 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:50:33.719 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:50:33.719 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:50:33.719 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:50:33.719 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:0 2024-09-12T06:50:33.719 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:50:33.719 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:50:33.719 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:50:33.719 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:0 2024-09-12T06:50:33.719 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:50:33.719 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:50:33.719 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:50:33.719 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:50:33.720 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T06:50:33.720 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:50:33.720 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:50:33.720 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=0 2024-09-12T06:50:33.720 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=0 2024-09-12T06:50:33.720 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:50:33.735 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:50:33.735 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:50:33.735 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:50:33.735 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_46 2024-09-12T06:50:33.825 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2024-09-12T06:50:33.825 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T06:50:33.825 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:admin_socket: invalid command 2024-09-12T06:50:33.829 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:50:33.829 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_46 2024-09-12T06:50:33.829 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:1 2024-09-12T06:50:33.829 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:50:33.829 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:50:33.829 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T06:50:33.829 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:1 2024-09-12T06:50:33.830 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:50:33.830 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:50:33.830 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:50:33.830 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:50:33.830 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T06:50:33.830 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:50:33.830 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:50:33.830 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=1 2024-09-12T06:50:33.830 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:50:33.830 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T06:50:33.830 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:1 2024-09-12T06:50:33.830 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:50:33.830 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:50:33.830 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T06:50:33.831 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:1 2024-09-12T06:50:33.831 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:50:33.831 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:50:33.831 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:50:33.831 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:50:33.831 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T06:50:33.831 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:50:33.831 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:50:33.831 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=1 2024-09-12T06:50:33.831 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=1 2024-09-12T06:50:33.831 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T06:50:33.847 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:50:33.847 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:50:33.847 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:50:33.847 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_46 2024-09-12T06:50:33.945 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ return 0 2024-09-12T06:50:33.946 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T06:50:33.946 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo ' "state": "Replaying"' 2024-09-12T06:50:33.946 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep -i Replaying 2024-09-12T06:50:33.946 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: "state": "Replaying" 2024-09-12T06:50:33.948 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_state=started 2024-09-12T06:50:33.948 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test started = started 2024-09-12T06:50:33.948 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:50:33.949 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_46 2024-09-12T06:50:33.949 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:50:33.949 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:50:33.949 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:50:33.949 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:50:33.949 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_46 2024-09-12T06:50:33.949 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' journal = journal ']' 2024-09-12T06:50:33.949 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror image_46 2024-09-12T06:50:33.949 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:50:33.949 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:50:33.949 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:50:33.949 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:50:33.949 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_46 2024-09-12T06:50:33.950 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-12T06:50:33.950 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-12T06:50:33.950 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:50:33.950 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:50:33.950 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:50:34.157 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror image_46 2024-09-12T06:50:34.157 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:50:34.157 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:34.157 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_46 2024-09-12T06:50:34.158 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:50:34.158 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n image_46 ']' 2024-09-12T06:50:34.158 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/image_46' 2024-09-12T06:50:34.158 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_46 2024-09-12T06:50:34.158 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:50:34.158 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:50:34.158 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:50:34.159 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:50:34.159 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:50:34.159 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:50:34.159 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:50:34.159 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:50:34.159 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:50:34.159 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:50:34.160 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_46 2024-09-12T06:50:34.160 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:50:34.160 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:50:34.160 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:50:34.160 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:50:34.160 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:50:34.161 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:50:34.161 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:50:34.161 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:50:34.161 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:50:34.161 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:50:34.161 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:50:34.161 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:50:34.162 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:50:34.162 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:50:34.162 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:50:34.162 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:50:34.162 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:50:34.162 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:50:34.162 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:50:34.162 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:50:34.162 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:50:34.162 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:50:34.162 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:50:34.162 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:50:34.163 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:50:34.163 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:50:34.163 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:50:34.163 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:50:34.163 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:50:34.163 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:50:34.171 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:50:34.171 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:50:34.171 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:50:34.171 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_46 2024-09-12T06:50:34.259 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2024-09-12T06:50:34.259 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd mirror flush : flush rbd mirror 2024-09-12T06:50:34.259 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:admin_socket: invalid command 2024-09-12T06:50:34.263 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:50:34.263 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_46 2024-09-12T06:50:34.263 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:1 2024-09-12T06:50:34.263 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:50:34.263 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:50:34.263 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-12T06:50:34.263 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:1 2024-09-12T06:50:34.263 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:50:34.263 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:50:34.264 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:50:34.264 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=1 2024-09-12T06:50:34.264 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-12T06:50:34.264 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:50:34.264 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:50:34.264 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=1 2024-09-12T06:50:34.264 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:50:34.264 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-12T06:50:34.264 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:1 2024-09-12T06:50:34.264 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:50:34.265 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:50:34.265 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T06:50:34.265 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:1 2024-09-12T06:50:34.265 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:50:34.266 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:50:34.266 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:50:34.266 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:50:34.266 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T06:50:34.266 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:50:34.266 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:50:34.266 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=1 2024-09-12T06:50:34.266 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:50:34.266 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T06:50:34.281 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:50:34.281 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:50:34.282 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:50:34.282 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_46 2024-09-12T06:50:34.381 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:50:34.381 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror image_46 2024-09-12T06:50:34.381 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:50:34.381 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:50:34.381 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_46 2024-09-12T06:50:34.381 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_46 '' 2024-09-12T06:50:34.381 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:50:34.381 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:50:34.381 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_46 2024-09-12T06:50:34.381 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:50:34.382 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_46.status 2024-09-12T06:50:34.382 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_46.status 2024-09-12T06:50:34.382 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:50:34.382 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_46.status 2024-09-12T06:50:34.382 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_46 2024-09-12T06:50:34.382 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_46.status 2024-09-12T06:50:34.428 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 3 2024-09-12T06:50:34.429 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 3 2024-09-12T06:50:34.429 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:50:34.429 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=12711], [object_number=14, tag_tid=1, entry_tid=12710]]], state=connected] 2024-09-12T06:50:34.429 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=12711], [object_number=14, tag_tid=1, entry_tid=12710]]], state=connected] 2024-09-12T06:50:34.432 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_46.status 2024-09-12T06:50:34.433 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-09-12T06:50:34.434 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror image_46 2024-09-12T06:50:34.435 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:50:34.435 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:50:34.435 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_46 2024-09-12T06:50:34.435 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_46 '..*' 2024-09-12T06:50:34.435 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:50:34.435 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:50:34.435 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_46 2024-09-12T06:50:34.435 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:50:34.435 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_46.status 2024-09-12T06:50:34.435 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_46.status 2024-09-12T06:50:34.435 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:50:34.436 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_46.status 2024-09-12T06:50:34.436 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_46 2024-09-12T06:50:34.436 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_46.status 2024-09-12T06:50:34.482 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 3 2024-09-12T06:50:34.482 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 3 2024-09-12T06:50:34.482 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:50:34.482 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=12711], [object_number=14, tag_tid=1, entry_tid=12710]]], state=connected] 2024-09-12T06:50:34.482 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=12711], [object_number=14, tag_tid=1, entry_tid=12710]]], state=connected] 2024-09-12T06:50:34.485 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_46.status 2024-09-12T06:50:34.487 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-09-12T06:50:34.487 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=1' 2024-09-12T06:50:34.487 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:50:34.487 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_snap_present cluster1 mirror image_46 snap 2024-09-12T06:50:34.487 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:50:34.487 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:34.487 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_46 2024-09-12T06:50:34.488 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:50:34.488 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:50:34.488 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T06:50:34.488 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:50:35.488 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 info mirror/image_46@snap 2024-09-12T06:50:35.532 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:rbd image 'image_46': 2024-09-12T06:50:35.533 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: size 128 MiB in 32 objects 2024-09-12T06:50:35.533 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: order 22 (4 MiB objects) 2024-09-12T06:50:35.533 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: snapshot_count: 1 2024-09-12T06:50:35.533 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: id: 10af87d7e25b 2024-09-12T06:50:35.533 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: block_name_prefix: rbd_data.10af87d7e25b 2024-09-12T06:50:35.533 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: format: 2 2024-09-12T06:50:35.533 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: features: layering, exclusive-lock, journaling 2024-09-12T06:50:35.534 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: op_features: 2024-09-12T06:50:35.534 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: flags: 2024-09-12T06:50:35.534 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: create_timestamp: Thu Sep 12 06:44:50 2024 2024-09-12T06:50:35.534 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: access_timestamp: Thu Sep 12 06:44:50 2024 2024-09-12T06:50:35.534 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: modify_timestamp: Thu Sep 12 06:45:51 2024 2024-09-12T06:50:35.534 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: protected: False 2024-09-12T06:50:35.534 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: journal: 10af87d7e25b 2024-09-12T06:50:35.534 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring state: enabled 2024-09-12T06:50:35.534 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring mode: journal 2024-09-12T06:50:35.534 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring global id: cb9a1c7f-d774-4fc2-8ba6-87d6de519ed7 2024-09-12T06:50:35.534 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring primary: false 2024-09-12T06:50:35.536 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:50:35.537 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror image_46 snap 2024-09-12T06:50:35.537 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:35.537 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_46 2024-09-12T06:50:35.537 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:50:35.537 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:50:35.537 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_46.export 2024-09-12T06:50:35.537 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_46.export 2024-09-12T06:50:35.537 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_46.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_46.export 2024-09-12T06:50:35.537 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export image_46@snap /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_46.export 2024-09-12T06:50:35.698 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:50:35.705 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export image_46@snap /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_46.export 2024-09-12T06:50:35.926 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:50:35.932 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_46.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_46.export 2024-09-12T06:50:35.986 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_46.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_46.export 2024-09-12T06:50:36.020 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:50:36.020 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:50:36.020 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_47 2024-09-12T06:50:36.020 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_snap cluster2 mirror image_47 snap 2024-09-12T06:50:36.020 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:50:36.020 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:36.020 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_47 2024-09-12T06:50:36.020 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:50:36.021 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror snap create image_47@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T06:50:36.716 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_started cluster1 mirror image_47 2024-09-12T06:50:36.717 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:50:36.717 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:36.717 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_47 2024-09-12T06:50:36.717 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_state cluster1 mirror image_47 started 2024-09-12T06:50:36.717 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:50:36.717 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:36.717 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_47 2024-09-12T06:50:36.717 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=started 2024-09-12T06:50:36.717 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:50:36.717 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:50:36.717 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:50:37.718 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_replay_state cluster1 mirror image_47 started 2024-09-12T06:50:37.718 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:50:37.718 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:37.718 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_47 2024-09-12T06:50:37.718 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=started 2024-09-12T06:50:37.718 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local status_result 2024-09-12T06:50:37.718 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=stopped 2024-09-12T06:50:37.719 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_47 2024-09-12T06:50:37.719 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_instance=cluster1 2024-09-12T06:50:37.719 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:50:37.720 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -i state 2024-09-12T06:50:37.720 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:50:37.720 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance=cluster1 2024-09-12T06:50:37.720 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local loop_instance 2024-09-12T06:50:37.720 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:50:37.720 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sleep 0 2024-09-12T06:50:37.720 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:50:37.720 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ seq 0 1 2024-09-12T06:50:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:50:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_47 2024-09-12T06:50:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:0 2024-09-12T06:50:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:50:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:50:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:50:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:50:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:50:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:50:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:50:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:50:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:50:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:50:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:50:37.724 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:50:37.725 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:50:37.725 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:50:37.725 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:0 2024-09-12T06:50:37.725 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:50:37.725 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:50:37.725 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:50:37.725 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:0 2024-09-12T06:50:37.725 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:50:37.725 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:50:37.725 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:50:37.725 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:50:37.725 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T06:50:37.725 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:50:37.725 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:50:37.726 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=0 2024-09-12T06:50:37.726 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=0 2024-09-12T06:50:37.726 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:50:37.746 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:50:37.746 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:50:37.747 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:50:37.747 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_47 2024-09-12T06:50:37.855 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ return 0 2024-09-12T06:50:37.855 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T06:50:37.855 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo ' "state": "Replaying"' 2024-09-12T06:50:37.855 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep -i Replaying 2024-09-12T06:50:37.856 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: "state": "Replaying" 2024-09-12T06:50:37.858 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_state=started 2024-09-12T06:50:37.859 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test started = started 2024-09-12T06:50:37.859 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:50:37.859 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_47 2024-09-12T06:50:37.859 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:50:37.859 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:50:37.859 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:50:37.859 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:50:37.859 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_47 2024-09-12T06:50:37.859 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' journal = journal ']' 2024-09-12T06:50:37.859 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror image_47 2024-09-12T06:50:37.859 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:50:37.860 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:50:37.860 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:50:37.860 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:50:37.860 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_47 2024-09-12T06:50:37.860 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-12T06:50:37.860 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-12T06:50:37.860 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:50:37.860 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:50:37.860 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:50:38.058 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror image_47 2024-09-12T06:50:38.059 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:50:38.059 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:38.059 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_47 2024-09-12T06:50:38.059 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:50:38.059 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n image_47 ']' 2024-09-12T06:50:38.059 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/image_47' 2024-09-12T06:50:38.059 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_47 2024-09-12T06:50:38.059 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:50:38.059 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:50:38.059 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:50:38.059 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:50:38.059 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:50:38.059 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:50:38.060 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:50:38.060 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:50:38.060 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:50:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:50:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_47 2024-09-12T06:50:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:50:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:50:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:50:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:50:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:50:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:50:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:50:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:50:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:50:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:50:38.064 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:50:38.064 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:50:38.064 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:50:38.064 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:50:38.064 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:50:38.064 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:50:38.064 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:50:38.064 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:50:38.064 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:50:38.064 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:50:38.064 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:50:38.064 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:50:38.064 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:50:38.065 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:50:38.065 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:50:38.065 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:50:38.065 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:50:38.065 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:50:38.065 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:50:38.065 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:50:38.081 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:50:38.081 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:50:38.081 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:50:38.081 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_47 2024-09-12T06:50:38.181 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:50:38.181 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror image_47 2024-09-12T06:50:38.182 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:50:38.182 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:50:38.182 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_47 2024-09-12T06:50:38.182 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_47 '' 2024-09-12T06:50:38.182 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:50:38.182 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:50:38.182 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_47 2024-09-12T06:50:38.183 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:50:38.183 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_47.status 2024-09-12T06:50:38.183 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_47.status 2024-09-12T06:50:38.183 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:50:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_47.status 2024-09-12T06:50:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_47 2024-09-12T06:50:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_47.status 2024-09-12T06:50:38.229 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 1 2024-09-12T06:50:38.229 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 1 2024-09-12T06:50:38.229 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:50:38.229 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6559], [object_number=6, tag_tid=1, entry_tid=6558]]], state=connected] 2024-09-12T06:50:38.229 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6559], [object_number=6, tag_tid=1, entry_tid=6558]]], state=connected] 2024-09-12T06:50:38.232 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_47.status 2024-09-12T06:50:38.233 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-12T06:50:38.235 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror image_47 2024-09-12T06:50:38.235 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:50:38.235 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:50:38.235 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_47 2024-09-12T06:50:38.235 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_47 '..*' 2024-09-12T06:50:38.235 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:50:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:50:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_47 2024-09-12T06:50:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:50:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_47.status 2024-09-12T06:50:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_47.status 2024-09-12T06:50:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:50:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_47.status 2024-09-12T06:50:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_47 2024-09-12T06:50:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_47.status 2024-09-12T06:50:38.282 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 1 2024-09-12T06:50:38.283 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 1 2024-09-12T06:50:38.283 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:50:38.283 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6559], [object_number=6, tag_tid=1, entry_tid=6558]]], state=connected] 2024-09-12T06:50:38.283 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6559], [object_number=6, tag_tid=1, entry_tid=6558]]], state=connected] 2024-09-12T06:50:38.285 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_47.status 2024-09-12T06:50:38.288 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-12T06:50:38.288 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-09-12T06:50:38.288 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:50:38.288 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_snap_present cluster1 mirror image_47 snap 2024-09-12T06:50:38.288 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:50:38.288 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:38.288 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_47 2024-09-12T06:50:38.288 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:50:38.288 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:50:38.288 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T06:50:38.289 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:50:39.288 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 info mirror/image_47@snap 2024-09-12T06:50:39.336 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:rbd image 'image_47': 2024-09-12T06:50:39.337 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: size 128 MiB in 32 objects 2024-09-12T06:50:39.337 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: order 22 (4 MiB objects) 2024-09-12T06:50:39.337 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: snapshot_count: 1 2024-09-12T06:50:39.337 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: id: 10a8a537f0a1 2024-09-12T06:50:39.337 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: block_name_prefix: rbd_data.10a8a537f0a1 2024-09-12T06:50:39.337 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: format: 2 2024-09-12T06:50:39.337 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: features: layering, exclusive-lock, journaling 2024-09-12T06:50:39.337 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: op_features: 2024-09-12T06:50:39.337 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: flags: 2024-09-12T06:50:39.337 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: create_timestamp: Thu Sep 12 06:45:06 2024 2024-09-12T06:50:39.337 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: access_timestamp: Thu Sep 12 06:45:06 2024 2024-09-12T06:50:39.337 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: modify_timestamp: Thu Sep 12 06:46:07 2024 2024-09-12T06:50:39.338 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: protected: False 2024-09-12T06:50:39.338 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: journal: 10a8a537f0a1 2024-09-12T06:50:39.338 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring state: enabled 2024-09-12T06:50:39.338 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring mode: journal 2024-09-12T06:50:39.338 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring global id: 90fb8873-8dc6-4a06-821a-084c54d93839 2024-09-12T06:50:39.338 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring primary: false 2024-09-12T06:50:39.340 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:50:39.340 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror image_47 snap 2024-09-12T06:50:39.340 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:39.340 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_47 2024-09-12T06:50:39.340 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:50:39.340 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:50:39.340 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_47.export 2024-09-12T06:50:39.341 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_47.export 2024-09-12T06:50:39.341 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_47.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_47.export 2024-09-12T06:50:39.341 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export image_47@snap /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_47.export 2024-09-12T06:50:39.496 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:50:39.503 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export image_47@snap /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_47.export 2024-09-12T06:50:39.615 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:50:39.617+0000 7f80c298c640 -1 bdev(0x556b0184ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 1 2024-09-12T06:50:39.616 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-12T06:50:39.618+0000 7f80bf185640 -1 bdev(0x556b0184ec00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 3 2024-09-12T06:50:39.692 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:50:39.697 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_47.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_47.export 2024-09-12T06:50:39.752 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_47.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_47.export 2024-09-12T06:50:39.791 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:50:39.791 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:50:39.791 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_48 2024-09-12T06:50:39.791 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_snap cluster2 mirror image_48 snap 2024-09-12T06:50:39.792 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:50:39.792 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:39.792 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_48 2024-09-12T06:50:39.792 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:50:39.792 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror snap create image_48@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T06:50:42.734 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_started cluster1 mirror image_48 2024-09-12T06:50:42.734 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:50:42.734 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:42.734 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_48 2024-09-12T06:50:42.734 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_state cluster1 mirror image_48 started 2024-09-12T06:50:42.734 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:50:42.734 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:42.734 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_48 2024-09-12T06:50:42.734 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=started 2024-09-12T06:50:42.734 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:50:42.734 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:50:42.734 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:50:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_replay_state cluster1 mirror image_48 started 2024-09-12T06:50:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:50:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_48 2024-09-12T06:50:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=started 2024-09-12T06:50:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local status_result 2024-09-12T06:50:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=stopped 2024-09-12T06:50:43.736 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_48 2024-09-12T06:50:43.736 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_instance=cluster1 2024-09-12T06:50:43.736 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:50:43.736 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:50:43.736 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -i state 2024-09-12T06:50:43.736 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance=cluster1 2024-09-12T06:50:43.736 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local loop_instance 2024-09-12T06:50:43.736 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:50:43.736 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sleep 0 2024-09-12T06:50:43.738 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:50:43.738 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ seq 0 1 2024-09-12T06:50:43.738 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:50:43.738 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_48 2024-09-12T06:50:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:0 2024-09-12T06:50:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:50:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:50:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:50:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:50:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:50:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:50:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:50:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:50:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:50:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:50:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:50:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:50:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:50:43.740 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:50:43.740 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:0 2024-09-12T06:50:43.741 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:50:43.741 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:50:43.741 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:50:43.741 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:0 2024-09-12T06:50:43.741 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:50:43.742 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:50:43.742 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:50:43.742 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:50:43.742 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T06:50:43.742 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:50:43.742 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:50:43.742 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=0 2024-09-12T06:50:43.742 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=0 2024-09-12T06:50:43.742 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:50:43.759 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:50:43.759 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:50:43.759 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:50:43.759 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_48 2024-09-12T06:50:43.850 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2024-09-12T06:50:43.850 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T06:50:43.850 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:admin_socket: invalid command 2024-09-12T06:50:43.853 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:50:43.854 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_48 2024-09-12T06:50:43.854 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:1 2024-09-12T06:50:43.854 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:50:43.854 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:50:43.854 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T06:50:43.854 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:1 2024-09-12T06:50:43.854 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:50:43.854 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:50:43.854 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:50:43.854 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:50:43.854 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T06:50:43.854 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:50:43.855 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:50:43.855 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=1 2024-09-12T06:50:43.855 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:50:43.855 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T06:50:43.855 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:1 2024-09-12T06:50:43.855 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:50:43.855 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:50:43.855 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T06:50:43.855 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:1 2024-09-12T06:50:43.855 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:50:43.855 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:50:43.855 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:50:43.855 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:50:43.855 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T06:50:43.856 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:50:43.856 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:50:43.856 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=1 2024-09-12T06:50:43.856 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=1 2024-09-12T06:50:43.856 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T06:50:43.872 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:50:43.872 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:50:43.872 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:50:43.872 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_48 2024-09-12T06:50:43.971 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ return 0 2024-09-12T06:50:43.972 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T06:50:43.972 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo ' "state": "Replaying"' 2024-09-12T06:50:43.972 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep -i Replaying 2024-09-12T06:50:43.972 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: "state": "Replaying" 2024-09-12T06:50:43.974 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_state=started 2024-09-12T06:50:43.974 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test started = started 2024-09-12T06:50:43.974 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:50:43.974 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_48 2024-09-12T06:50:43.975 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:50:43.975 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:50:43.975 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:50:43.975 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:50:43.975 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_48 2024-09-12T06:50:43.975 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' journal = journal ']' 2024-09-12T06:50:43.975 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror image_48 2024-09-12T06:50:43.975 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:50:43.975 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:50:43.975 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:50:43.975 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:50:43.975 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_48 2024-09-12T06:50:43.975 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-12T06:50:43.975 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-12T06:50:43.976 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:50:43.976 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:50:43.976 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:50:44.175 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror image_48 2024-09-12T06:50:44.175 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:50:44.175 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:44.176 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_48 2024-09-12T06:50:44.176 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:50:44.176 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n image_48 ']' 2024-09-12T06:50:44.176 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/image_48' 2024-09-12T06:50:44.176 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_48 2024-09-12T06:50:44.176 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:50:44.176 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:50:44.176 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:50:44.176 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:50:44.176 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:50:44.176 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:50:44.176 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:50:44.176 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:50:44.177 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:50:44.179 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:50:44.179 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_48 2024-09-12T06:50:44.179 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:50:44.180 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:50:44.180 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:50:44.180 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:50:44.180 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:50:44.180 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:50:44.180 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:50:44.180 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:50:44.180 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:50:44.180 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:50:44.180 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:50:44.180 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:50:44.180 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:50:44.180 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:50:44.181 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:50:44.181 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:50:44.181 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:50:44.181 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:50:44.181 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:50:44.181 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:50:44.181 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:50:44.181 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:50:44.181 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:50:44.181 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:50:44.181 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:50:44.181 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:50:44.181 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:50:44.182 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:50:44.182 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:50:44.182 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:50:44.197 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:50:44.197 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:50:44.197 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:50:44.197 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_48 2024-09-12T06:50:44.285 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2024-09-12T06:50:44.285 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd mirror flush : flush rbd mirror 2024-09-12T06:50:44.285 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:admin_socket: invalid command 2024-09-12T06:50:44.288 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:50:44.288 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_48 2024-09-12T06:50:44.288 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:1 2024-09-12T06:50:44.288 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:50:44.288 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:50:44.289 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-12T06:50:44.289 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:1 2024-09-12T06:50:44.289 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:50:44.289 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:50:44.289 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:50:44.289 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=1 2024-09-12T06:50:44.289 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-12T06:50:44.289 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:50:44.289 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:50:44.289 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=1 2024-09-12T06:50:44.289 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:50:44.289 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-12T06:50:44.289 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:1 2024-09-12T06:50:44.290 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:50:44.290 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:50:44.290 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T06:50:44.290 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:1 2024-09-12T06:50:44.290 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:50:44.290 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:50:44.290 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:50:44.290 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:50:44.290 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T06:50:44.290 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:50:44.290 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:50:44.290 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=1 2024-09-12T06:50:44.290 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:50:44.291 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T06:50:44.307 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:50:44.308 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:50:44.308 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:50:44.308 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_48 2024-09-12T06:50:44.403 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:50:44.404 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror image_48 2024-09-12T06:50:44.404 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:50:44.404 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:50:44.404 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_48 2024-09-12T06:50:44.404 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_48 '' 2024-09-12T06:50:44.404 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:50:44.404 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:50:44.404 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_48 2024-09-12T06:50:44.404 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:50:44.404 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_48.status 2024-09-12T06:50:44.404 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_48.status 2024-09-12T06:50:44.405 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:50:44.405 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_48.status 2024-09-12T06:50:44.405 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_48 2024-09-12T06:50:44.406 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_48.status 2024-09-12T06:50:44.451 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 3 2024-09-12T06:50:44.451 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 3 2024-09-12T06:50:44.451 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:50:44.451 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=12495], [object_number=14, tag_tid=1, entry_tid=12494]]], state=connected] 2024-09-12T06:50:44.451 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=13, tag_tid=1, entry_tid=12497], [object_number=12, tag_tid=1, entry_tid=12496], [object_number=15, tag_tid=1, entry_tid=12495], [object_number=14, tag_tid=1, entry_tid=12494]]], state=connected] 2024-09-12T06:50:44.453 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_48.status 2024-09-12T06:50:44.456 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-09-12T06:50:44.456 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror image_48 2024-09-12T06:50:44.456 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:50:44.456 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:50:44.456 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_48 2024-09-12T06:50:44.456 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_48 '..*' 2024-09-12T06:50:44.456 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:50:44.457 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:50:44.457 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_48 2024-09-12T06:50:44.457 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:50:44.457 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_48.status 2024-09-12T06:50:44.457 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_48.status 2024-09-12T06:50:44.457 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:50:44.457 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_48.status 2024-09-12T06:50:44.458 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_48 2024-09-12T06:50:44.458 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_48.status 2024-09-12T06:50:44.503 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 3 2024-09-12T06:50:44.503 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 3 2024-09-12T06:50:44.503 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:50:44.503 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=12495], [object_number=14, tag_tid=1, entry_tid=12494]]], state=connected] 2024-09-12T06:50:44.503 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=13, tag_tid=1, entry_tid=12497], [object_number=12, tag_tid=1, entry_tid=12496], [object_number=15, tag_tid=1, entry_tid=12495], [object_number=14, tag_tid=1, entry_tid=12494]]], state=connected] 2024-09-12T06:50:44.505 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_48.status 2024-09-12T06:50:44.508 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=13, tag_tid=1, entry_tid=12497' 2024-09-12T06:50:44.508 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=13, tag_tid=1, entry_tid=12497' 2024-09-12T06:50:44.508 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 'object_number=13, tag_tid=1, entry_tid=12497' '!=' '' 2024-09-12T06:50:44.508 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ break 2024-09-12T06:50:44.508 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 'object_number=13, tag_tid=1, entry_tid=12497' = '' 2024-09-12T06:50:44.508 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ last_mirror_pos='object_number=13, tag_tid=1, entry_tid=12497' 2024-09-12T06:50:44.508 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=1' 2024-09-12T06:50:44.509 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-12T06:50:44.509 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:50:44.509 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_tag=2 2024-09-12T06:50:44.509 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=13, tag_tid=1, entry_tid=12497' 2024-09-12T06:50:44.510 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-12T06:50:44.510 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:50:44.510 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_tag=1 2024-09-12T06:50:44.511 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=1' 2024-09-12T06:50:44.511 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-12T06:50:44.511 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:50:44.512 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_entry=1 2024-09-12T06:50:44.512 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo 'object_number=13, tag_tid=1, entry_tid=12497' 2024-09-12T06:50:44.513 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-12T06:50:44.513 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut -d= -f 2 2024-09-12T06:50:44.513 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_entry=12497 2024-09-12T06:50:44.514 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 2 = 1 -a 1 -le 12497 2024-09-12T06:50:44.514 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:50:44.514 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:50:44.514 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:50:44.716 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror image_48 2024-09-12T06:50:44.716 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:50:44.716 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:44.716 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_48 2024-09-12T06:50:44.716 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:50:44.716 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n image_48 ']' 2024-09-12T06:50:44.716 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/image_48' 2024-09-12T06:50:44.716 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_48 2024-09-12T06:50:44.717 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:50:44.717 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:50:44.717 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:50:44.717 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:50:44.717 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:50:44.717 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:50:44.717 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:50:44.717 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:50:44.717 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:50:44.720 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:50:44.720 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_48 2024-09-12T06:50:44.720 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:50:44.720 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:50:44.720 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:50:44.720 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:50:44.721 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:50:44.721 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:50:44.721 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:50:44.721 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:50:44.721 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:50:44.721 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:50:44.721 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:50:44.721 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:50:44.721 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:50:44.721 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:50:44.721 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:50:44.721 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:50:44.721 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:50:44.722 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:50:44.722 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:50:44.722 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:50:44.722 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:50:44.722 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:50:44.722 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:50:44.722 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:50:44.722 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:50:44.722 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:50:44.722 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:50:44.722 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:50:44.722 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:50:44.722 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:50:44.737 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:50:44.737 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:50:44.737 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:50:44.737 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_48 2024-09-12T06:50:44.799 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:50:44.800+0000 7fac12a52640 -1 reset not still connected to 0x55749aea7790 2024-09-12T06:50:44.799 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:50:44.800+0000 7fac12a52640 -1 reset not still connected to 0x55749b1cbc70 2024-09-12T06:50:44.803 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:50:44.805+0000 7fac12a52640 -1 reset not still connected to 0x55749b2bf930 2024-09-12T06:50:44.846 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2024-09-12T06:50:44.847 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd mirror flush : flush rbd mirror 2024-09-12T06:50:44.847 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:admin_socket: invalid command 2024-09-12T06:50:44.850 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:50:44.850 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_48 2024-09-12T06:50:44.850 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:1 2024-09-12T06:50:44.850 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:50:44.850 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:50:44.851 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-12T06:50:44.851 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:1 2024-09-12T06:50:44.851 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:50:44.851 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:50:44.851 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:50:44.851 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=1 2024-09-12T06:50:44.851 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-12T06:50:44.851 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:50:44.851 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:50:44.851 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=1 2024-09-12T06:50:44.851 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:50:44.851 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-12T06:50:44.851 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:1 2024-09-12T06:50:44.852 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:50:44.852 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:50:44.852 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T06:50:44.852 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:1 2024-09-12T06:50:44.852 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:50:44.852 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:50:44.852 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:50:44.852 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:50:44.852 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T06:50:44.852 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:50:44.852 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:50:44.852 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=1 2024-09-12T06:50:44.852 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:50:44.852 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T06:50:44.869 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:50:44.869 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:50:44.870 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:50:44.870 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_48 2024-09-12T06:50:44.977 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:50:44.978 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror image_48 2024-09-12T06:50:44.978 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:50:44.978 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:50:44.978 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_48 2024-09-12T06:50:44.978 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_48 '' 2024-09-12T06:50:44.978 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:50:44.978 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:50:44.979 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_48 2024-09-12T06:50:44.979 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:50:44.979 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_48.status 2024-09-12T06:50:44.979 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_48.status 2024-09-12T06:50:44.979 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:50:44.980 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_48.status 2024-09-12T06:50:44.980 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_48 2024-09-12T06:50:44.980 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_48.status 2024-09-12T06:50:45.028 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 3 2024-09-12T06:50:45.028 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 3 2024-09-12T06:50:45.028 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:50:45.028 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=12495], [object_number=14, tag_tid=1, entry_tid=12494]]], state=connected] 2024-09-12T06:50:45.028 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=13, tag_tid=1, entry_tid=12497], [object_number=12, tag_tid=1, entry_tid=12496], [object_number=15, tag_tid=1, entry_tid=12495], [object_number=14, tag_tid=1, entry_tid=12494]]], state=connected] 2024-09-12T06:50:45.031 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_48.status 2024-09-12T06:50:45.032 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-09-12T06:50:45.034 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror image_48 2024-09-12T06:50:45.034 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:50:45.034 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:50:45.034 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_48 2024-09-12T06:50:45.034 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_48 '..*' 2024-09-12T06:50:45.034 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:50:45.035 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:50:45.035 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_48 2024-09-12T06:50:45.035 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:50:45.035 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_48.status 2024-09-12T06:50:45.035 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_48.status 2024-09-12T06:50:45.035 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:50:45.036 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_48.status 2024-09-12T06:50:45.036 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_48 2024-09-12T06:50:45.036 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_48.status 2024-09-12T06:50:45.080 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 3 2024-09-12T06:50:45.080 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 3 2024-09-12T06:50:45.081 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:50:45.081 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=12495], [object_number=14, tag_tid=1, entry_tid=12494]]], state=connected] 2024-09-12T06:50:45.081 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=13, tag_tid=1, entry_tid=12497], [object_number=12, tag_tid=1, entry_tid=12496], [object_number=15, tag_tid=1, entry_tid=12495], [object_number=14, tag_tid=1, entry_tid=12494]]], state=connected] 2024-09-12T06:50:45.083 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_48.status 2024-09-12T06:50:45.086 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=13, tag_tid=1, entry_tid=12497' 2024-09-12T06:50:45.086 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=13, tag_tid=1, entry_tid=12497' 2024-09-12T06:50:45.086 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 'object_number=13, tag_tid=1, entry_tid=12497' '!=' 'object_number=13, tag_tid=1, entry_tid=12497' 2024-09-12T06:50:45.086 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:50:45.086 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.4 2024-09-12T06:50:45.487 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror image_48 2024-09-12T06:50:45.488 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:50:45.488 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:45.488 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_48 2024-09-12T06:50:45.488 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:50:45.488 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n image_48 ']' 2024-09-12T06:50:45.488 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/image_48' 2024-09-12T06:50:45.488 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_48 2024-09-12T06:50:45.488 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:50:45.489 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:50:45.489 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:50:45.489 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:50:45.489 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:50:45.489 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:50:45.489 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:50:45.489 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:50:45.489 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:50:45.492 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:50:45.493 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_48 2024-09-12T06:50:45.493 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:50:45.493 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:50:45.493 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:50:45.493 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:50:45.493 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:50:45.493 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:50:45.493 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:50:45.493 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:50:45.493 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:50:45.494 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:50:45.494 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:50:45.494 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:50:45.494 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:50:45.494 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:50:45.494 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:50:45.494 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:50:45.494 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:50:45.494 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:50:45.494 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:50:45.494 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:50:45.494 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:50:45.494 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:50:45.495 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:50:45.495 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:50:45.495 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:50:45.495 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:50:45.495 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:50:45.495 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:50:45.495 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:50:45.495 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:50:45.509 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:50:45.509 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:50:45.510 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:50:45.510 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_48 2024-09-12T06:50:45.595 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2024-09-12T06:50:45.595 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd mirror flush : flush rbd mirror 2024-09-12T06:50:45.595 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:admin_socket: invalid command 2024-09-12T06:50:45.599 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:50:45.599 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_48 2024-09-12T06:50:45.599 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:1 2024-09-12T06:50:45.599 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:50:45.599 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:50:45.599 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-12T06:50:45.599 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:1 2024-09-12T06:50:45.599 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:50:45.599 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:50:45.599 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:50:45.599 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=1 2024-09-12T06:50:45.600 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-12T06:50:45.600 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:50:45.600 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:50:45.600 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=1 2024-09-12T06:50:45.600 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:50:45.600 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-12T06:50:45.600 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:1 2024-09-12T06:50:45.600 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:50:45.600 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:50:45.600 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T06:50:45.600 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:1 2024-09-12T06:50:45.600 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:50:45.600 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:50:45.600 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:50:45.600 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:50:45.601 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T06:50:45.601 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:50:45.601 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:50:45.601 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=1 2024-09-12T06:50:45.601 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:50:45.601 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T06:50:45.616 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:50:45.616 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:50:45.617 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:50:45.617 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_48 2024-09-12T06:50:45.714 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:50:45.715 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror image_48 2024-09-12T06:50:45.715 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:50:45.715 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:50:45.715 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_48 2024-09-12T06:50:45.715 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_48 '' 2024-09-12T06:50:45.715 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:50:45.715 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:50:45.715 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_48 2024-09-12T06:50:45.715 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:50:45.715 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_48.status 2024-09-12T06:50:45.715 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_48.status 2024-09-12T06:50:45.716 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:50:45.716 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_48.status 2024-09-12T06:50:45.716 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_48 2024-09-12T06:50:45.716 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_48.status 2024-09-12T06:50:45.761 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 3 2024-09-12T06:50:45.762 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 3 2024-09-12T06:50:45.762 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:50:45.762 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=12495], [object_number=14, tag_tid=1, entry_tid=12494]]], state=connected] 2024-09-12T06:50:45.762 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=12495], [object_number=14, tag_tid=1, entry_tid=12494]]], state=connected] 2024-09-12T06:50:45.764 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_48.status 2024-09-12T06:50:45.766 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-09-12T06:50:45.768 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror image_48 2024-09-12T06:50:45.768 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:50:45.768 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:50:45.768 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_48 2024-09-12T06:50:45.769 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_48 '..*' 2024-09-12T06:50:45.769 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:50:45.769 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:50:45.769 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_48 2024-09-12T06:50:45.769 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:50:45.769 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_48.status 2024-09-12T06:50:45.769 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_48.status 2024-09-12T06:50:45.769 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:50:45.769 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_48.status 2024-09-12T06:50:45.769 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_48 2024-09-12T06:50:45.769 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_48.status 2024-09-12T06:50:45.814 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 3 2024-09-12T06:50:45.814 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 3 2024-09-12T06:50:45.814 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:50:45.814 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=12495], [object_number=14, tag_tid=1, entry_tid=12494]]], state=connected] 2024-09-12T06:50:45.815 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=12495], [object_number=14, tag_tid=1, entry_tid=12494]]], state=connected] 2024-09-12T06:50:45.817 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_48.status 2024-09-12T06:50:45.819 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-09-12T06:50:45.819 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=1' 2024-09-12T06:50:45.820 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:50:45.820 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_snap_present cluster1 mirror image_48 snap 2024-09-12T06:50:45.820 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:50:45.820 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:45.820 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_48 2024-09-12T06:50:45.820 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:50:45.820 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:50:45.820 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T06:50:45.820 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:50:46.820 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 info mirror/image_48@snap 2024-09-12T06:50:46.869 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:rbd image 'image_48': 2024-09-12T06:50:46.869 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: size 128 MiB in 32 objects 2024-09-12T06:50:46.869 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: order 22 (4 MiB objects) 2024-09-12T06:50:46.869 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: snapshot_count: 1 2024-09-12T06:50:46.869 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: id: 10af7b4185ae 2024-09-12T06:50:46.869 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: block_name_prefix: rbd_data.10af7b4185ae 2024-09-12T06:50:46.869 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: format: 2 2024-09-12T06:50:46.869 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: features: layering, exclusive-lock, journaling 2024-09-12T06:50:46.869 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: op_features: 2024-09-12T06:50:46.869 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: flags: 2024-09-12T06:50:46.869 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: create_timestamp: Thu Sep 12 06:45:12 2024 2024-09-12T06:50:46.869 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: access_timestamp: Thu Sep 12 06:45:12 2024 2024-09-12T06:50:46.870 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: modify_timestamp: Thu Sep 12 06:46:13 2024 2024-09-12T06:50:46.870 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: protected: False 2024-09-12T06:50:46.870 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: journal: 10af7b4185ae 2024-09-12T06:50:46.870 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring state: enabled 2024-09-12T06:50:46.870 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring mode: journal 2024-09-12T06:50:46.870 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring global id: 856866e7-ed8e-491f-ac6e-f649e965a31b 2024-09-12T06:50:46.870 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring primary: false 2024-09-12T06:50:46.873 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:50:46.873 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror image_48 snap 2024-09-12T06:50:46.873 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:46.873 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_48 2024-09-12T06:50:46.873 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:50:46.873 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:50:46.873 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_48.export 2024-09-12T06:50:46.873 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_48.export 2024-09-12T06:50:46.873 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_48.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_48.export 2024-09-12T06:50:46.874 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export image_48@snap /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_48.export 2024-09-12T06:50:47.039 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:50:47.045 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export image_48@snap /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_48.export 2024-09-12T06:50:47.297 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:50:47.305 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_48.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_48.export 2024-09-12T06:50:47.361 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_48.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_48.export 2024-09-12T06:50:47.393 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:50:47.393 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:50:47.393 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_49 2024-09-12T06:50:47.393 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_snap cluster2 mirror image_49 snap 2024-09-12T06:50:47.394 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:50:47.394 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:47.394 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_49 2024-09-12T06:50:47.394 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:50:47.394 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror snap create image_49@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T06:50:49.746 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_started cluster1 mirror image_49 2024-09-12T06:50:49.746 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:50:49.746 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:49.746 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_49 2024-09-12T06:50:49.746 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_state cluster1 mirror image_49 started 2024-09-12T06:50:49.746 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:50:49.747 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:49.747 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_49 2024-09-12T06:50:49.747 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=started 2024-09-12T06:50:49.747 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:50:49.747 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:50:49.747 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:50:50.747 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_replay_state cluster1 mirror image_49 started 2024-09-12T06:50:50.748 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:50:50.748 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:50.748 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_49 2024-09-12T06:50:50.748 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=started 2024-09-12T06:50:50.748 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local status_result 2024-09-12T06:50:50.748 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=stopped 2024-09-12T06:50:50.748 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_49 2024-09-12T06:50:50.748 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_instance=cluster1 2024-09-12T06:50:50.748 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:50:50.748 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -i state 2024-09-12T06:50:50.748 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:50:50.748 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance=cluster1 2024-09-12T06:50:50.748 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local loop_instance 2024-09-12T06:50:50.749 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:50:50.749 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sleep 0 2024-09-12T06:50:50.752 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:50:50.752 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ seq 0 1 2024-09-12T06:50:50.753 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:50:50.753 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_49 2024-09-12T06:50:50.753 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:0 2024-09-12T06:50:50.753 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:50:50.753 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:50:50.753 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:50:50.753 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:50:50.753 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:50:50.753 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:50:50.754 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:50:50.754 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:50:50.754 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:50:50.754 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:50:50.754 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:50:50.754 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:50:50.758 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:50:50.758 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:50:50.758 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:0 2024-09-12T06:50:50.758 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:50:50.758 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:50:50.758 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:50:50.758 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:0 2024-09-12T06:50:50.758 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:50:50.759 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:50:50.759 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:50:50.759 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:50:50.759 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T06:50:50.759 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:50:50.759 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:50:50.759 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=0 2024-09-12T06:50:50.759 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=0 2024-09-12T06:50:50.759 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:50:50.769 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:50:50.769 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:50:50.769 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:50:50.769 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_49 2024-09-12T06:50:50.870 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ return 0 2024-09-12T06:50:50.870 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T06:50:50.871 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo ' "state": "Replaying"' 2024-09-12T06:50:50.871 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep -i Replaying 2024-09-12T06:50:50.871 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: "state": "Replaying" 2024-09-12T06:50:50.874 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_state=started 2024-09-12T06:50:50.874 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test started = started 2024-09-12T06:50:50.874 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:50:50.874 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_49 2024-09-12T06:50:50.874 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:50:50.874 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:50:50.875 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:50:50.875 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:50:50.875 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_49 2024-09-12T06:50:50.875 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' journal = journal ']' 2024-09-12T06:50:50.875 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror image_49 2024-09-12T06:50:50.875 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:50:50.875 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:50:50.875 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:50:50.875 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:50:50.876 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_49 2024-09-12T06:50:50.876 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-12T06:50:50.876 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-12T06:50:50.876 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:50:50.876 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:50:50.876 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:50:51.080 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror image_49 2024-09-12T06:50:51.080 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:50:51.081 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:51.081 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_49 2024-09-12T06:50:51.081 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:50:51.081 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n image_49 ']' 2024-09-12T06:50:51.081 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/image_49' 2024-09-12T06:50:51.081 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_49 2024-09-12T06:50:51.081 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:50:51.081 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:50:51.081 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:50:51.081 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:50:51.081 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:50:51.081 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:50:51.081 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:50:51.082 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:50:51.082 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:50:51.082 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:50:51.082 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_49 2024-09-12T06:50:51.082 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:50:51.082 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:50:51.082 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:50:51.082 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:50:51.082 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:50:51.082 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:50:51.082 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:50:51.082 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:50:51.082 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:50:51.083 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:50:51.083 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:50:51.083 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:50:51.083 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:50:51.083 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:50:51.083 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:50:51.083 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:50:51.083 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:50:51.083 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:50:51.083 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:50:51.083 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:50:51.083 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:50:51.083 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:50:51.084 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:50:51.084 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:50:51.084 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:50:51.084 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:50:51.084 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:50:51.084 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:50:51.084 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:50:51.084 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:50:51.096 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:50:51.096 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:50:51.096 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:50:51.096 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_49 2024-09-12T06:50:51.196 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:50:51.196 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror image_49 2024-09-12T06:50:51.196 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:50:51.196 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:50:51.197 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_49 2024-09-12T06:50:51.197 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_49 '' 2024-09-12T06:50:51.197 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:50:51.197 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:50:51.197 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_49 2024-09-12T06:50:51.198 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:50:51.198 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_49.status 2024-09-12T06:50:51.198 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_49.status 2024-09-12T06:50:51.198 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:50:51.198 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_49.status 2024-09-12T06:50:51.198 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_49 2024-09-12T06:50:51.199 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_49.status 2024-09-12T06:50:51.242 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 3 2024-09-12T06:50:51.242 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 3 2024-09-12T06:50:51.242 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:50:51.242 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=11879], [object_number=14, tag_tid=1, entry_tid=11878]]], state=connected] 2024-09-12T06:50:51.242 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=11879], [object_number=14, tag_tid=1, entry_tid=11878]]], state=connected] 2024-09-12T06:50:51.245 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_49.status 2024-09-12T06:50:51.247 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-09-12T06:50:51.248 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror image_49 2024-09-12T06:50:51.248 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:50:51.248 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:50:51.248 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_49 2024-09-12T06:50:51.248 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_49 '..*' 2024-09-12T06:50:51.248 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:50:51.248 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:50:51.249 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_49 2024-09-12T06:50:51.249 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:50:51.249 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_49.status 2024-09-12T06:50:51.249 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_49.status 2024-09-12T06:50:51.249 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:50:51.249 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_49.status 2024-09-12T06:50:51.249 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_49 2024-09-12T06:50:51.250 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_49.status 2024-09-12T06:50:51.295 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 3 2024-09-12T06:50:51.296 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 3 2024-09-12T06:50:51.296 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:50:51.296 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=11879], [object_number=14, tag_tid=1, entry_tid=11878]]], state=connected] 2024-09-12T06:50:51.296 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=11879], [object_number=14, tag_tid=1, entry_tid=11878]]], state=connected] 2024-09-12T06:50:51.298 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_49.status 2024-09-12T06:50:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-09-12T06:50:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=1' 2024-09-12T06:50:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:50:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_snap_present cluster1 mirror image_49 snap 2024-09-12T06:50:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:50:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_49 2024-09-12T06:50:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:50:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:50:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T06:50:51.302 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:50:52.302 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 info mirror/image_49@snap 2024-09-12T06:50:52.350 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:rbd image 'image_49': 2024-09-12T06:50:52.350 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: size 128 MiB in 32 objects 2024-09-12T06:50:52.351 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: order 22 (4 MiB objects) 2024-09-12T06:50:52.351 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: snapshot_count: 1 2024-09-12T06:50:52.351 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: id: 10a84485bfaa 2024-09-12T06:50:52.351 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: block_name_prefix: rbd_data.10a84485bfaa 2024-09-12T06:50:52.351 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: format: 2 2024-09-12T06:50:52.351 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: features: layering, exclusive-lock, journaling 2024-09-12T06:50:52.351 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: op_features: 2024-09-12T06:50:52.351 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: flags: 2024-09-12T06:50:52.351 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: create_timestamp: Thu Sep 12 06:45:22 2024 2024-09-12T06:50:52.351 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: access_timestamp: Thu Sep 12 06:45:22 2024 2024-09-12T06:50:52.351 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: modify_timestamp: Thu Sep 12 06:46:23 2024 2024-09-12T06:50:52.351 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: protected: False 2024-09-12T06:50:52.351 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: journal: 10a84485bfaa 2024-09-12T06:50:52.351 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring state: enabled 2024-09-12T06:50:52.352 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring mode: journal 2024-09-12T06:50:52.352 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring global id: d035c8da-83d3-42c0-8618-78dfa9264fe7 2024-09-12T06:50:52.352 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring primary: false 2024-09-12T06:50:52.354 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:50:52.354 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror image_49 snap 2024-09-12T06:50:52.354 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:52.354 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_49 2024-09-12T06:50:52.354 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:50:52.354 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:50:52.354 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_49.export 2024-09-12T06:50:52.355 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_49.export 2024-09-12T06:50:52.355 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_49.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_49.export 2024-09-12T06:50:52.355 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export image_49@snap /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_49.export 2024-09-12T06:50:52.519 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:50:52.526 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export image_49@snap /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_49.export 2024-09-12T06:50:52.715 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:50:52.717+0000 7fa3126a7640 -1 bdev(0x55f858298c00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-09-12T06:50:52.775 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:50:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_49.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_49.export 2024-09-12T06:50:52.836 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_49.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_49.export 2024-09-12T06:50:52.868 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:50:52.868 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:50:52.868 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_50 2024-09-12T06:50:52.868 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_snap cluster2 mirror image_50 snap 2024-09-12T06:50:52.868 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:50:52.868 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:52.868 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_50 2024-09-12T06:50:52.868 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:50:52.868 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror snap create image_50@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T06:50:53.767 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_started cluster1 mirror image_50 2024-09-12T06:50:53.767 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:50:53.767 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:53.767 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_50 2024-09-12T06:50:53.767 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_replay_state cluster1 mirror image_50 started 2024-09-12T06:50:53.768 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:50:53.768 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:53.768 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_50 2024-09-12T06:50:53.768 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=started 2024-09-12T06:50:53.768 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:50:53.768 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:50:53.768 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:50:54.769 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_replay_state cluster1 mirror image_50 started 2024-09-12T06:50:54.769 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:50:54.769 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:54.769 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_50 2024-09-12T06:50:54.769 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=started 2024-09-12T06:50:54.770 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local status_result 2024-09-12T06:50:54.770 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=stopped 2024-09-12T06:50:54.770 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_50 2024-09-12T06:50:54.770 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -i state 2024-09-12T06:50:54.770 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_instance=cluster1 2024-09-12T06:50:54.770 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:50:54.770 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:50:54.770 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance=cluster1 2024-09-12T06:50:54.770 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local loop_instance 2024-09-12T06:50:54.770 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:50:54.770 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sleep 0 2024-09-12T06:50:54.772 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:50:54.772 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ seq 0 1 2024-09-12T06:50:54.772 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:50:54.772 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_50 2024-09-12T06:50:54.772 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:0 2024-09-12T06:50:54.772 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:50:54.772 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:50:54.772 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:50:54.773 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:50:54.773 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:50:54.773 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:50:54.773 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:50:54.773 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:50:54.773 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:50:54.775 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:50:54.775 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:50:54.776 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:50:54.776 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:50:54.776 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:50:54.776 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:0 2024-09-12T06:50:54.776 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:50:54.776 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:50:54.776 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:50:54.776 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:0 2024-09-12T06:50:54.776 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:50:54.776 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:50:54.776 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:50:54.776 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:50:54.776 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T06:50:54.777 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:50:54.777 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:50:54.777 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=0 2024-09-12T06:50:54.777 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=0 2024-09-12T06:50:54.777 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:50:54.800 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:50:54.800 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:50:54.800 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:50:54.800 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_50 2024-09-12T06:50:54.904 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2024-09-12T06:50:54.904 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T06:50:54.904 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:admin_socket: invalid command 2024-09-12T06:50:54.907 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:50:54.907 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_50 2024-09-12T06:50:54.908 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:1 2024-09-12T06:50:54.908 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ shift 2024-09-12T06:50:54.908 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:50:54.908 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T06:50:54.908 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:1 2024-09-12T06:50:54.908 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:50:54.908 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:50:54.908 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:50:54.908 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:50:54.908 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T06:50:54.908 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:50:54.908 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:50:54.909 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=1 2024-09-12T06:50:54.909 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:50:54.909 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T06:50:54.909 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local local_cluster=cluster1:1 2024-09-12T06:50:54.909 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1 2024-09-12T06:50:54.909 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:50:54.909 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T06:50:54.909 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:1 2024-09-12T06:50:54.909 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T06:50:54.910 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:50:54.910 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:50:54.910 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:50:54.910 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T06:50:54.910 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval local_cluster=cluster1 2024-09-12T06:50:54.910 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ local_cluster=cluster1 2024-09-12T06:50:54.910 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=1 2024-09-12T06:50:54.910 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=1 2024-09-12T06:50:54.910 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T06:50:54.926 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:50:54.926 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:50:54.927 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:50:54.927 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_50 2024-09-12T06:50:55.029 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ return 0 2024-09-12T06:50:55.030 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T06:50:55.030 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo ' "state": "Replaying"' 2024-09-12T06:50:55.030 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ grep -i Replaying 2024-09-12T06:50:55.031 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: "state": "Replaying" 2024-09-12T06:50:55.033 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_state=started 2024-09-12T06:50:55.033 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test started = started 2024-09-12T06:50:55.033 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:50:55.033 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_50 2024-09-12T06:50:55.033 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:50:55.033 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:50:55.033 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:50:55.034 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:50:55.034 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_50 2024-09-12T06:50:55.034 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' journal = journal ']' 2024-09-12T06:50:55.034 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror mirror image_50 2024-09-12T06:50:55.034 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_cluster=cluster1 2024-09-12T06:50:55.034 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:50:55.034 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local local_pool=mirror 2024-09-12T06:50:55.034 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local remote_pool=mirror 2024-09-12T06:50:55.034 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_50 2024-09-12T06:50:55.034 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-12T06:50:55.034 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-12T06:50:55.034 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:50:55.034 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T06:50:55.035 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.2 2024-09-12T06:50:55.240 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ flush cluster1 mirror image_50 2024-09-12T06:50:55.240 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:50:55.240 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:55.240 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_50 2024-09-12T06:50:55.240 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local 'cmd=rbd mirror flush' 2024-09-12T06:50:55.240 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n image_50 ']' 2024-09-12T06:50:55.241 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmd='rbd mirror flush mirror/image_50' 2024-09-12T06:50:55.241 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_50 2024-09-12T06:50:55.241 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_instance=cluster1 2024-09-12T06:50:55.241 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:50:55.241 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:50:55.241 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance=cluster1 2024-09-12T06:50:55.242 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loop_instance 2024-09-12T06:50:55.242 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T06:50:55.242 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:50:55.242 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-12T06:50:55.242 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:50:55.242 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:50:55.242 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_50 2024-09-12T06:50:55.243 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:50:55.243 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:50:55.243 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:50:55.243 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:50:55.243 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:0 2024-09-12T06:50:55.243 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:50:55.244 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:50:55.244 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:50:55.244 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:50:55.244 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T06:50:55.244 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:50:55.244 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:50:55.244 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:50:55.245 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:50:55.245 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:50:55.245 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:50:55.245 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:50:55.245 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:50:55.245 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:50:55.246 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:50:55.246 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:50:55.246 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:50:55.246 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:50:55.246 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:50:55.246 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:50:55.246 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:50:55.246 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:50:55.247 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:50:55.247 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:50:55.247 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:50:55.257 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:50:55.257 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:50:55.257 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:50:55.257 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_50 2024-09-12T06:50:55.344 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2024-09-12T06:50:55.344 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd mirror flush : flush rbd mirror 2024-09-12T06:50:55.344 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:admin_socket: invalid command 2024-09-12T06:50:55.349 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:50:55.350 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_50 2024-09-12T06:50:55.350 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:1 2024-09-12T06:50:55.350 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:50:55.350 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:50:55.350 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-12T06:50:55.350 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1:1 2024-09-12T06:50:55.350 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:50:55.350 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:50:55.350 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:50:55.350 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=1 2024-09-12T06:50:55.350 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-12T06:50:55.350 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:50:55.351 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:50:55.351 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=1 2024-09-12T06:50:55.351 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:50:55.351 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-12T06:50:55.351 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:1 2024-09-12T06:50:55.351 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:50:55.351 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:50:55.351 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T06:50:55.351 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:1 2024-09-12T06:50:55.351 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:50:55.351 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:50:55.351 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:50:55.351 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:50:55.351 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T06:50:55.352 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:50:55.352 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:50:55.352 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=1 2024-09-12T06:50:55.352 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:50:55.352 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T06:50:55.365 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:50:55.365 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:50:55.366 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:50:55.366 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_50 2024-09-12T06:50:55.463 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:50:55.463 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_master_journal_position cluster2 mirror image_50 2024-09-12T06:50:55.463 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:50:55.463 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:50:55.464 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_50 2024-09-12T06:50:55.464 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_50 '' 2024-09-12T06:50:55.464 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:50:55.464 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:50:55.464 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_50 2024-09-12T06:50:55.464 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local id_regexp= 2024-09-12T06:50:55.464 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_50.status 2024-09-12T06:50:55.464 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_50.status 2024-09-12T06:50:55.464 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:50:55.465 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_50.status 2024-09-12T06:50:55.465 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_50 2024-09-12T06:50:55.465 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_50.status 2024-09-12T06:50:55.510 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 1 2024-09-12T06:50:55.510 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 1 2024-09-12T06:50:55.510 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:50:55.510 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=4990], [object_number=7, tag_tid=1, entry_tid=4987]]], state=connected] 2024-09-12T06:50:55.510 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=4990], [object_number=7, tag_tid=1, entry_tid=4987]]], state=connected] 2024-09-12T06:50:55.513 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_50.status 2024-09-12T06:50:55.515 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-12T06:50:55.515 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_mirror_journal_position cluster2 mirror image_50 2024-09-12T06:50:55.516 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:50:55.516 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:50:55.516 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_50 2024-09-12T06:50:55.516 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_journal_position cluster2 mirror image_50 '..*' 2024-09-12T06:50:55.516 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:50:55.516 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:50:55.516 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_50 2024-09-12T06:50:55.516 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local 'id_regexp=..*' 2024-09-12T06:50:55.516 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ mkfname cluster2-mirror-image_50.status 2024-09-12T06:50:55.516 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo cluster2-mirror-image_50.status 2024-09-12T06:50:55.517 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-12T06:50:55.517 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local status_log=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_50.status 2024-09-12T06:50:55.517 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_50 2024-09-12T06:50:55.517 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ tee /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_50.status 2024-09-12T06:50:55.562 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:minimum_set: 1 2024-09-12T06:50:55.562 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:active_set: 1 2024-09-12T06:50:55.562 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:registered clients: 2024-09-12T06:50:55.562 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=4990], [object_number=7, tag_tid=1, entry_tid=4987]]], state=connected] 2024-09-12T06:50:55.562 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: [id=840e8a4e-26a4-4837-a52c-3ad55abc3e4c, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=4990], [object_number=7, tag_tid=1, entry_tid=4987]]], state=connected] 2024-09-12T06:50:55.565 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_50.status 2024-09-12T06:50:55.568 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-12T06:50:55.568 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-09-12T06:50:55.568 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:50:55.568 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_snap_present cluster1 mirror image_50 snap 2024-09-12T06:50:55.568 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:50:55.569 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:55.569 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_50 2024-09-12T06:50:55.569 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:50:55.569 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:50:55.569 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T06:50:55.569 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:50:56.568 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 info mirror/image_50@snap 2024-09-12T06:50:56.613 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:rbd image 'image_50': 2024-09-12T06:50:56.613 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: size 128 MiB in 32 objects 2024-09-12T06:50:56.613 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: order 22 (4 MiB objects) 2024-09-12T06:50:56.613 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: snapshot_count: 1 2024-09-12T06:50:56.613 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: id: 10af3008f1ce 2024-09-12T06:50:56.614 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: block_name_prefix: rbd_data.10af3008f1ce 2024-09-12T06:50:56.614 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: format: 2 2024-09-12T06:50:56.614 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: features: layering, exclusive-lock, journaling 2024-09-12T06:50:56.614 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: op_features: 2024-09-12T06:50:56.614 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: flags: 2024-09-12T06:50:56.614 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: create_timestamp: Thu Sep 12 06:45:35 2024 2024-09-12T06:50:56.614 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: access_timestamp: Thu Sep 12 06:45:35 2024 2024-09-12T06:50:56.614 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: modify_timestamp: Thu Sep 12 06:46:36 2024 2024-09-12T06:50:56.614 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: protected: False 2024-09-12T06:50:56.614 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: journal: 10af3008f1ce 2024-09-12T06:50:56.614 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring state: enabled 2024-09-12T06:50:56.614 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring mode: journal 2024-09-12T06:50:56.614 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring global id: 76384d82-50e4-4d56-adbd-290fe8ca4238 2024-09-12T06:50:56.614 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring primary: false 2024-09-12T06:50:56.617 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:50:56.617 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ compare_image_snaps mirror image_50 snap 2024-09-12T06:50:56.617 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:56.617 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_50 2024-09-12T06:50:56.618 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap_name=snap 2024-09-12T06:50:56.618 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local ret=0 2024-09-12T06:50:56.618 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local rmt_export=/tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_50.export 2024-09-12T06:50:56.618 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local loc_export=/tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_50.export 2024-09-12T06:50:56.618 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_50.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_50.export 2024-09-12T06:50:56.618 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 -p mirror export image_50@snap /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_50.export 2024-09-12T06:50:56.765 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:50:56.772 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster1 -p mirror export image_50@snap /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_50.export 2024-09-12T06:50:56.957 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T06:50:56.962 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cmp /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_50.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_50.export 2024-09-12T06:50:57.018 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /tmp/tmp.GzjHZGs9Z0/cluster2-mirror-image_50.export /tmp/tmp.GzjHZGs9Z0/cluster1-mirror-image_50.export 2024-09-12T06:50:57.052 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:50:57.052 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ testlog 'TEST: delete many images' 2024-09-12T06:50:57.052 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ tee -a /tmp/tmp.GzjHZGs9Z0/rbd-mirror.test.log 2024-09-12T06:50:57.052 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ date '+%F %T' 2024-09-12T06:50:57.053 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo 2024-09-12 06:50:57 TEST: delete many images 2024-09-12T06:50:57.053 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:2024-09-12 06:50:57 TEST: delete many images 2024-09-12T06:50:57.053 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 1 50 2024-09-12T06:50:57.054 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:50:57.054 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_1 2024-09-12T06:50:57.055 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_snapshot cluster2 mirror image_1 snap 2024-09-12T06:50:57.055 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:50:57.055 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:57.055 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_1 2024-09-12T06:50:57.055 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap=snap 2024-09-12T06:50:57.055 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 snap rm mirror/image_1@snap 2024-09-12T06:50:58.040 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing snap: 100% complete...done. 2024-09-12T06:50:58.053 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image_retry cluster2 mirror image_1 2024-09-12T06:50:58.053 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:50:58.053 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:58.053 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_1 2024-09-12T06:50:58.053 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T06:50:58.053 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:50:58.054 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image cluster2 mirror image_1 2024-09-12T06:50:58.054 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:50:58.054 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:58.054 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_1 2024-09-12T06:50:58.054 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2024-09-12T06:50:58.102 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2024-09-12T06:50:58.330 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T06:50:58.334 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:50:58.334 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:50:58.334 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_2 2024-09-12T06:50:58.334 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_snapshot cluster2 mirror image_2 snap 2024-09-12T06:50:58.334 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:50:58.334 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:58.334 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_2 2024-09-12T06:50:58.335 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap=snap 2024-09-12T06:50:58.335 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 snap rm mirror/image_2@snap 2024-09-12T06:50:59.054 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing snap: 100% complete...done. 2024-09-12T06:50:59.076 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image_retry cluster2 mirror image_2 2024-09-12T06:50:59.076 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:50:59.076 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:59.076 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_2 2024-09-12T06:50:59.076 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T06:50:59.076 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:50:59.077 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image cluster2 mirror image_2 2024-09-12T06:50:59.077 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:50:59.077 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:59.077 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_2 2024-09-12T06:50:59.077 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2024-09-12T06:50:59.124 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2024-09-12T06:50:59.311 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T06:50:59.315 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:50:59.315 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:50:59.315 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_3 2024-09-12T06:50:59.316 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_snapshot cluster2 mirror image_3 snap 2024-09-12T06:50:59.316 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:50:59.316 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:50:59.316 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_3 2024-09-12T06:50:59.316 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap=snap 2024-09-12T06:50:59.316 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 snap rm mirror/image_3@snap 2024-09-12T06:51:00.042 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing snap: 100% complete...done. 2024-09-12T06:51:00.055 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image_retry cluster2 mirror image_3 2024-09-12T06:51:00.055 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:00.055 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:00.055 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_3 2024-09-12T06:51:00.055 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T06:51:00.055 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:51:00.056 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image cluster2 mirror image_3 2024-09-12T06:51:00.056 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:00.056 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:00.056 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_3 2024-09-12T06:51:00.056 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2024-09-12T06:51:00.104 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2024-09-12T06:51:00.305 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T06:51:00.309 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:00.309 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:00.309 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_4 2024-09-12T06:51:00.309 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_snapshot cluster2 mirror image_4 snap 2024-09-12T06:51:00.309 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:00.309 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:00.309 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_4 2024-09-12T06:51:00.309 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap=snap 2024-09-12T06:51:00.309 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 snap rm mirror/image_4@snap 2024-09-12T06:51:01.047 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing snap: 100% complete...done. 2024-09-12T06:51:01.062 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image_retry cluster2 mirror image_4 2024-09-12T06:51:01.063 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:01.063 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:01.063 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_4 2024-09-12T06:51:01.063 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T06:51:01.063 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:51:01.064 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image cluster2 mirror image_4 2024-09-12T06:51:01.064 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:01.064 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:01.064 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_4 2024-09-12T06:51:01.064 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2024-09-12T06:51:01.112 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2024-09-12T06:51:01.303 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T06:51:01.307 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:01.307 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:01.307 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_5 2024-09-12T06:51:01.308 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_snapshot cluster2 mirror image_5 snap 2024-09-12T06:51:01.308 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:01.308 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:01.308 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_5 2024-09-12T06:51:01.308 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap=snap 2024-09-12T06:51:01.308 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 snap rm mirror/image_5@snap 2024-09-12T06:51:02.048 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing snap: 100% complete...done. 2024-09-12T06:51:02.061 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image_retry cluster2 mirror image_5 2024-09-12T06:51:02.062 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:02.062 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:02.062 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_5 2024-09-12T06:51:02.062 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T06:51:02.062 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:51:02.062 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image cluster2 mirror image_5 2024-09-12T06:51:02.063 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:02.063 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:02.063 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_5 2024-09-12T06:51:02.063 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2024-09-12T06:51:02.110 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2024-09-12T06:51:02.314 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T06:51:02.317 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:02.317 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:02.317 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_6 2024-09-12T06:51:02.318 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_snapshot cluster2 mirror image_6 snap 2024-09-12T06:51:02.318 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:02.318 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:02.318 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_6 2024-09-12T06:51:02.318 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap=snap 2024-09-12T06:51:02.318 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 snap rm mirror/image_6@snap 2024-09-12T06:51:03.057 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing snap: 100% complete...done. 2024-09-12T06:51:03.072 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image_retry cluster2 mirror image_6 2024-09-12T06:51:03.072 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:03.072 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:03.072 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_6 2024-09-12T06:51:03.072 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T06:51:03.072 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:51:03.073 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image cluster2 mirror image_6 2024-09-12T06:51:03.073 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:03.073 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:03.073 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_6 2024-09-12T06:51:03.074 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2024-09-12T06:51:03.120 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2024-09-12T06:51:03.311 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T06:51:03.316 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:03.316 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:03.316 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_7 2024-09-12T06:51:03.316 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_snapshot cluster2 mirror image_7 snap 2024-09-12T06:51:03.316 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:03.316 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:03.316 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_7 2024-09-12T06:51:03.317 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap=snap 2024-09-12T06:51:03.317 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 snap rm mirror/image_7@snap 2024-09-12T06:51:04.067 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing snap: 100% complete...done. 2024-09-12T06:51:04.080 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image_retry cluster2 mirror image_7 2024-09-12T06:51:04.081 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:04.081 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:04.081 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_7 2024-09-12T06:51:04.081 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T06:51:04.081 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:51:04.082 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image cluster2 mirror image_7 2024-09-12T06:51:04.082 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:04.082 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:04.082 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_7 2024-09-12T06:51:04.082 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2024-09-12T06:51:04.129 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2024-09-12T06:51:04.338 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T06:51:04.342 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:04.342 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:04.342 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_8 2024-09-12T06:51:04.342 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_snapshot cluster2 mirror image_8 snap 2024-09-12T06:51:04.342 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:04.342 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:04.342 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_8 2024-09-12T06:51:04.342 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap=snap 2024-09-12T06:51:04.342 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 snap rm mirror/image_8@snap 2024-09-12T06:51:05.037 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing snap: 100% complete...done. 2024-09-12T06:51:05.049 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image_retry cluster2 mirror image_8 2024-09-12T06:51:05.049 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:05.049 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:05.049 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_8 2024-09-12T06:51:05.049 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T06:51:05.049 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:51:05.050 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image cluster2 mirror image_8 2024-09-12T06:51:05.050 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:05.050 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:05.050 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_8 2024-09-12T06:51:05.050 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2024-09-12T06:51:05.098 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2024-09-12T06:51:05.273 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T06:51:05.278 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:05.278 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:05.278 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_9 2024-09-12T06:51:05.278 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_snapshot cluster2 mirror image_9 snap 2024-09-12T06:51:05.278 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:05.278 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:05.278 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_9 2024-09-12T06:51:05.278 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap=snap 2024-09-12T06:51:05.278 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 snap rm mirror/image_9@snap 2024-09-12T06:51:06.070 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing snap: 100% complete...done. 2024-09-12T06:51:06.081 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image_retry cluster2 mirror image_9 2024-09-12T06:51:06.081 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:06.081 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:06.081 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_9 2024-09-12T06:51:06.082 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T06:51:06.082 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:51:06.082 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image cluster2 mirror image_9 2024-09-12T06:51:06.083 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:06.083 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:06.083 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_9 2024-09-12T06:51:06.083 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2024-09-12T06:51:06.129 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2024-09-12T06:51:06.319 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:51:06.319+0000 7fa325ae1640 -1 reset not still connected to 0x55f85a4fcd00 2024-09-12T06:51:06.322 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:51:06.324+0000 7fa325ae1640 -1 reset not still connected to 0x55f85a4fcdd0 2024-09-12T06:51:06.322 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:51:06.324+0000 7fa325ae1640 -1 reset not still connected to 0x55f85ac22750 2024-09-12T06:51:06.323 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T06:51:06.324 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:51:06.326+0000 7fa325ae1640 -1 reset not still connected to 0x55f85ba408f0 2024-09-12T06:51:06.324 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:51:06.326+0000 7fa325ae1640 -1 reset not still connected to 0x55f85bac3520 2024-09-12T06:51:06.325 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:51:06.326+0000 7fa325ae1640 -1 reset not still connected to 0x55f869fdc0d0 2024-09-12T06:51:06.325 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:51:06.326+0000 7fa325ae1640 -1 reset not still connected to 0x55f869fdc340 2024-09-12T06:51:06.325 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:51:06.326+0000 7fa325ae1640 -1 reset not still connected to 0x55f86ee2fe10 2024-09-12T06:51:06.325 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:51:06.326+0000 7fa325ae1640 -1 reset not still connected to 0x55f878a54410 2024-09-12T06:51:06.325 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:51:06.326+0000 7fa325ae1640 -1 reset not still connected to 0x55f88041e270 2024-09-12T06:51:06.325 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:51:06.326+0000 7fa325ae1640 -1 reset not still connected to 0x55f880b4e4e0 2024-09-12T06:51:06.325 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:51:06.326+0000 7fa325ae1640 -1 reset not still connected to 0x55f88417c410 2024-09-12T06:51:06.325 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:51:06.326+0000 7fa325ae1640 -1 reset not still connected to 0x55f884309c70 2024-09-12T06:51:06.325 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:51:06.326+0000 7fa325ae1640 -1 reset not still connected to 0x55f8849a11e0 2024-09-12T06:51:06.325 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:51:06.326+0000 7fa325ae1640 -1 reset not still connected to 0x55f8863b4340 2024-09-12T06:51:06.325 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:51:06.326+0000 7fa325ae1640 -1 reset not still connected to 0x55f8863b4680 2024-09-12T06:51:06.326 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:51:06.326+0000 7fa325ae1640 -1 reset not still connected to 0x55f8863b5790 2024-09-12T06:51:06.326 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:51:06.326+0000 7fa325ae1640 -1 reset not still connected to 0x55f88b2cd5f0 2024-09-12T06:51:06.326 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:51:06.326+0000 7fa325ae1640 -1 reset not still connected to 0x55f890e69a00 2024-09-12T06:51:06.326 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:51:06.326+0000 7fa325ae1640 -1 reset not still connected to 0x55f8926ce0d0 2024-09-12T06:51:06.326 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:51:06.326+0000 7fa325ae1640 -1 reset not still connected to 0x55f8926ce1a0 2024-09-12T06:51:06.326 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:51:06.326+0000 7fa325ae1640 -1 reset not still connected to 0x55f896243040 2024-09-12T06:51:06.328 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:06.328 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:06.328 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_10 2024-09-12T06:51:06.328 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_snapshot cluster2 mirror image_10 snap 2024-09-12T06:51:06.328 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:06.328 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:06.328 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_10 2024-09-12T06:51:06.328 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap=snap 2024-09-12T06:51:06.328 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 snap rm mirror/image_10@snap 2024-09-12T06:51:07.076 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing snap: 100% complete...done. 2024-09-12T06:51:07.089 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image_retry cluster2 mirror image_10 2024-09-12T06:51:07.089 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:07.089 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:07.089 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_10 2024-09-12T06:51:07.089 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T06:51:07.089 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:51:07.090 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image cluster2 mirror image_10 2024-09-12T06:51:07.090 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:07.090 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:07.090 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_10 2024-09-12T06:51:07.090 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2024-09-12T06:51:07.137 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2024-09-12T06:51:07.276 INFO:tasks.ceph.cluster2.osd.1.smithi152.stderr:2024-09-12T06:51:07.277+0000 7f33f9c90640 -1 reset not still connected to 0x55c73c372340 2024-09-12T06:51:07.278 INFO:tasks.ceph.cluster2.osd.1.smithi152.stderr:2024-09-12T06:51:07.280+0000 7f33f9c90640 -1 reset not still connected to 0x55c73c373110 2024-09-12T06:51:07.279 INFO:tasks.ceph.cluster2.osd.1.smithi152.stderr:2024-09-12T06:51:07.280+0000 7f33f9c90640 -1 reset not still connected to 0x55c73c3736c0 2024-09-12T06:51:07.332 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T06:51:07.335 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:07.336 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:07.336 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_11 2024-09-12T06:51:07.336 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_snapshot cluster2 mirror image_11 snap 2024-09-12T06:51:07.336 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:07.336 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:07.336 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_11 2024-09-12T06:51:07.336 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap=snap 2024-09-12T06:51:07.336 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 snap rm mirror/image_11@snap 2024-09-12T06:51:08.079 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing snap: 100% complete...done. 2024-09-12T06:51:08.092 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image_retry cluster2 mirror image_11 2024-09-12T06:51:08.092 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:08.092 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:08.092 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_11 2024-09-12T06:51:08.092 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T06:51:08.092 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:51:08.093 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image cluster2 mirror image_11 2024-09-12T06:51:08.093 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:08.093 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:08.093 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_11 2024-09-12T06:51:08.094 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_11 2024-09-12T06:51:08.141 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 rm mirror/image_11 2024-09-12T06:51:08.321 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T06:51:08.325 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:08.325 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:08.325 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_12 2024-09-12T06:51:08.326 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_snapshot cluster2 mirror image_12 snap 2024-09-12T06:51:08.326 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:08.326 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:08.326 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_12 2024-09-12T06:51:08.326 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap=snap 2024-09-12T06:51:08.326 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 snap rm mirror/image_12@snap 2024-09-12T06:51:09.089 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing snap: 100% complete...done. 2024-09-12T06:51:09.106 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image_retry cluster2 mirror image_12 2024-09-12T06:51:09.106 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:09.106 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:09.106 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_12 2024-09-12T06:51:09.107 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T06:51:09.107 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:51:09.107 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image cluster2 mirror image_12 2024-09-12T06:51:09.107 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:09.107 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:09.108 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_12 2024-09-12T06:51:09.108 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_12 2024-09-12T06:51:09.155 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 rm mirror/image_12 2024-09-12T06:51:09.340 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T06:51:09.344 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:09.344 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:09.344 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_13 2024-09-12T06:51:09.344 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_snapshot cluster2 mirror image_13 snap 2024-09-12T06:51:09.344 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:09.345 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:09.345 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_13 2024-09-12T06:51:09.345 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap=snap 2024-09-12T06:51:09.345 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 snap rm mirror/image_13@snap 2024-09-12T06:51:10.044 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing snap: 100% complete...done. 2024-09-12T06:51:10.057 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image_retry cluster2 mirror image_13 2024-09-12T06:51:10.057 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:10.057 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:10.057 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_13 2024-09-12T06:51:10.058 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T06:51:10.058 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:51:10.058 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image cluster2 mirror image_13 2024-09-12T06:51:10.058 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:10.058 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:10.059 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_13 2024-09-12T06:51:10.059 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_13 2024-09-12T06:51:10.112 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 rm mirror/image_13 2024-09-12T06:51:10.327 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T06:51:10.331 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:10.331 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:10.331 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_14 2024-09-12T06:51:10.331 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_snapshot cluster2 mirror image_14 snap 2024-09-12T06:51:10.331 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:10.331 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:10.331 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_14 2024-09-12T06:51:10.331 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap=snap 2024-09-12T06:51:10.331 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 snap rm mirror/image_14@snap 2024-09-12T06:51:11.093 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing snap: 100% complete...done. 2024-09-12T06:51:11.110 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image_retry cluster2 mirror image_14 2024-09-12T06:51:11.110 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:11.110 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:11.110 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_14 2024-09-12T06:51:11.110 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T06:51:11.110 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:51:11.111 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image cluster2 mirror image_14 2024-09-12T06:51:11.111 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:11.111 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:11.111 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_14 2024-09-12T06:51:11.111 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_14 2024-09-12T06:51:11.158 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 rm mirror/image_14 2024-09-12T06:51:11.925 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T06:51:11.930 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:11.930 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:11.930 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_15 2024-09-12T06:51:11.931 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_snapshot cluster2 mirror image_15 snap 2024-09-12T06:51:11.931 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:11.931 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:11.931 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_15 2024-09-12T06:51:11.931 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap=snap 2024-09-12T06:51:11.932 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 snap rm mirror/image_15@snap 2024-09-12T06:51:12.089 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing snap: 100% complete...done. 2024-09-12T06:51:12.106 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image_retry cluster2 mirror image_15 2024-09-12T06:51:12.106 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:12.106 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:12.106 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_15 2024-09-12T06:51:12.106 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T06:51:12.106 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:51:12.107 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image cluster2 mirror image_15 2024-09-12T06:51:12.107 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:12.108 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:12.108 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_15 2024-09-12T06:51:12.108 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_15 2024-09-12T06:51:12.154 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 rm mirror/image_15 2024-09-12T06:51:12.330 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T06:51:12.335 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:12.336 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:12.336 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_16 2024-09-12T06:51:12.336 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_snapshot cluster2 mirror image_16 snap 2024-09-12T06:51:12.336 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:12.336 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:12.336 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_16 2024-09-12T06:51:12.336 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap=snap 2024-09-12T06:51:12.336 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 snap rm mirror/image_16@snap 2024-09-12T06:51:13.091 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing snap: 100% complete...done. 2024-09-12T06:51:13.113 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image_retry cluster2 mirror image_16 2024-09-12T06:51:13.113 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:13.113 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:13.113 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_16 2024-09-12T06:51:13.113 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T06:51:13.114 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:51:13.114 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image cluster2 mirror image_16 2024-09-12T06:51:13.114 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:13.114 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:13.114 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_16 2024-09-12T06:51:13.115 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_16 2024-09-12T06:51:13.162 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 rm mirror/image_16 2024-09-12T06:51:13.339 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T06:51:13.343 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:13.344 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:13.344 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_17 2024-09-12T06:51:13.344 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_snapshot cluster2 mirror image_17 snap 2024-09-12T06:51:13.344 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:13.344 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:13.344 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_17 2024-09-12T06:51:13.344 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap=snap 2024-09-12T06:51:13.344 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 snap rm mirror/image_17@snap 2024-09-12T06:51:14.104 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing snap: 100% complete...done. 2024-09-12T06:51:14.116 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image_retry cluster2 mirror image_17 2024-09-12T06:51:14.116 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:14.116 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:14.116 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_17 2024-09-12T06:51:14.116 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T06:51:14.116 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:51:14.117 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image cluster2 mirror image_17 2024-09-12T06:51:14.117 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:14.117 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:14.117 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_17 2024-09-12T06:51:14.117 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_17 2024-09-12T06:51:14.166 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 rm mirror/image_17 2024-09-12T06:51:14.366 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T06:51:14.370 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:14.370 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:14.370 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_18 2024-09-12T06:51:14.370 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_snapshot cluster2 mirror image_18 snap 2024-09-12T06:51:14.370 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:14.371 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:14.371 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_18 2024-09-12T06:51:14.371 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap=snap 2024-09-12T06:51:14.371 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 snap rm mirror/image_18@snap 2024-09-12T06:51:15.039 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing snap: 100% complete...done. 2024-09-12T06:51:15.058 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image_retry cluster2 mirror image_18 2024-09-12T06:51:15.059 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:15.059 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:15.059 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_18 2024-09-12T06:51:15.059 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T06:51:15.059 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:51:15.061 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image cluster2 mirror image_18 2024-09-12T06:51:15.061 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:15.061 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:15.061 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_18 2024-09-12T06:51:15.061 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_18 2024-09-12T06:51:15.110 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 rm mirror/image_18 2024-09-12T06:51:15.308 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T06:51:15.311 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:15.311 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:15.311 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_19 2024-09-12T06:51:15.312 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_snapshot cluster2 mirror image_19 snap 2024-09-12T06:51:15.312 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:15.312 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:15.312 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_19 2024-09-12T06:51:15.312 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap=snap 2024-09-12T06:51:15.312 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 snap rm mirror/image_19@snap 2024-09-12T06:51:16.110 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing snap: 100% complete...done. 2024-09-12T06:51:16.124 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image_retry cluster2 mirror image_19 2024-09-12T06:51:16.124 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:16.124 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:16.124 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_19 2024-09-12T06:51:16.125 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T06:51:16.125 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:51:16.126 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image cluster2 mirror image_19 2024-09-12T06:51:16.126 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:16.126 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:16.126 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_19 2024-09-12T06:51:16.126 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_19 2024-09-12T06:51:16.175 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 rm mirror/image_19 2024-09-12T06:51:16.379 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T06:51:16.383 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:16.383 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:16.383 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_20 2024-09-12T06:51:16.383 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_snapshot cluster2 mirror image_20 snap 2024-09-12T06:51:16.383 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:16.384 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:16.384 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_20 2024-09-12T06:51:16.384 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap=snap 2024-09-12T06:51:16.384 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 snap rm mirror/image_20@snap 2024-09-12T06:51:17.107 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing snap: 100% complete...done. 2024-09-12T06:51:17.123 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image_retry cluster2 mirror image_20 2024-09-12T06:51:17.123 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:17.123 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:17.124 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_20 2024-09-12T06:51:17.124 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T06:51:17.124 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:51:17.124 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image cluster2 mirror image_20 2024-09-12T06:51:17.125 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:17.125 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:17.125 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_20 2024-09-12T06:51:17.125 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_20 2024-09-12T06:51:17.172 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 rm mirror/image_20 2024-09-12T06:51:17.368 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T06:51:17.372 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:17.372 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:17.372 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_21 2024-09-12T06:51:17.372 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_snapshot cluster2 mirror image_21 snap 2024-09-12T06:51:17.372 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:17.372 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:17.372 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_21 2024-09-12T06:51:17.372 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap=snap 2024-09-12T06:51:17.372 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 snap rm mirror/image_21@snap 2024-09-12T06:51:18.108 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing snap: 100% complete...done. 2024-09-12T06:51:18.121 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image_retry cluster2 mirror image_21 2024-09-12T06:51:18.121 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:18.121 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:18.121 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_21 2024-09-12T06:51:18.122 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T06:51:18.122 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:51:18.122 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image cluster2 mirror image_21 2024-09-12T06:51:18.122 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:18.122 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:18.122 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_21 2024-09-12T06:51:18.123 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_21 2024-09-12T06:51:18.170 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 rm mirror/image_21 2024-09-12T06:51:18.380 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T06:51:18.383 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:18.383 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:18.384 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_22 2024-09-12T06:51:18.384 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_snapshot cluster2 mirror image_22 snap 2024-09-12T06:51:18.384 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:18.384 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:18.384 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_22 2024-09-12T06:51:18.384 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap=snap 2024-09-12T06:51:18.384 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 snap rm mirror/image_22@snap 2024-09-12T06:51:19.118 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing snap: 100% complete...done. 2024-09-12T06:51:19.136 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image_retry cluster2 mirror image_22 2024-09-12T06:51:19.136 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:19.137 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:19.137 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_22 2024-09-12T06:51:19.137 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T06:51:19.137 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:51:19.138 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image cluster2 mirror image_22 2024-09-12T06:51:19.138 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:19.138 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:19.138 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_22 2024-09-12T06:51:19.138 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_22 2024-09-12T06:51:19.187 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 rm mirror/image_22 2024-09-12T06:51:19.369 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T06:51:19.373 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:19.373 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:19.373 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_23 2024-09-12T06:51:19.373 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_snapshot cluster2 mirror image_23 snap 2024-09-12T06:51:19.373 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:19.373 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:19.374 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_23 2024-09-12T06:51:19.374 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap=snap 2024-09-12T06:51:19.374 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 snap rm mirror/image_23@snap 2024-09-12T06:51:20.038 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing snap: 100% complete...done. 2024-09-12T06:51:20.050 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image_retry cluster2 mirror image_23 2024-09-12T06:51:20.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:20.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:20.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_23 2024-09-12T06:51:20.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T06:51:20.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:51:20.052 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image cluster2 mirror image_23 2024-09-12T06:51:20.052 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:20.052 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:20.052 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_23 2024-09-12T06:51:20.052 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_23 2024-09-12T06:51:20.099 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 rm mirror/image_23 2024-09-12T06:51:20.294 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T06:51:20.298 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:20.298 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:20.298 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_24 2024-09-12T06:51:20.298 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_snapshot cluster2 mirror image_24 snap 2024-09-12T06:51:20.298 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:20.298 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:20.298 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_24 2024-09-12T06:51:20.298 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap=snap 2024-09-12T06:51:20.298 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 snap rm mirror/image_24@snap 2024-09-12T06:51:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing snap: 100% complete...done. 2024-09-12T06:51:21.131 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image_retry cluster2 mirror image_24 2024-09-12T06:51:21.131 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:21.131 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:21.131 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_24 2024-09-12T06:51:21.131 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T06:51:21.131 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:51:21.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image cluster2 mirror image_24 2024-09-12T06:51:21.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:21.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:21.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_24 2024-09-12T06:51:21.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_24 2024-09-12T06:51:21.180 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 rm mirror/image_24 2024-09-12T06:51:21.371 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T06:51:21.374 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:21.374 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:21.374 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_25 2024-09-12T06:51:21.375 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_snapshot cluster2 mirror image_25 snap 2024-09-12T06:51:21.375 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:21.375 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:21.375 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_25 2024-09-12T06:51:21.375 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap=snap 2024-09-12T06:51:21.375 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 snap rm mirror/image_25@snap 2024-09-12T06:51:22.131 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing snap: 100% complete...done. 2024-09-12T06:51:22.145 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image_retry cluster2 mirror image_25 2024-09-12T06:51:22.145 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:22.146 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:22.146 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_25 2024-09-12T06:51:22.146 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T06:51:22.146 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:51:22.147 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image cluster2 mirror image_25 2024-09-12T06:51:22.147 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:22.147 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:22.147 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_25 2024-09-12T06:51:22.147 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_25 2024-09-12T06:51:22.196 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 rm mirror/image_25 2024-09-12T06:51:22.389 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T06:51:22.393 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:22.393 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:22.393 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_26 2024-09-12T06:51:22.393 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_snapshot cluster2 mirror image_26 snap 2024-09-12T06:51:22.393 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:22.393 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:22.393 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_26 2024-09-12T06:51:22.393 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap=snap 2024-09-12T06:51:22.393 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 snap rm mirror/image_26@snap 2024-09-12T06:51:23.128 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing snap: 100% complete...done. 2024-09-12T06:51:23.145 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image_retry cluster2 mirror image_26 2024-09-12T06:51:23.146 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:23.146 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:23.146 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_26 2024-09-12T06:51:23.146 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T06:51:23.146 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:51:23.146 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image cluster2 mirror image_26 2024-09-12T06:51:23.147 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:23.147 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:23.147 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_26 2024-09-12T06:51:23.147 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_26 2024-09-12T06:51:23.193 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 rm mirror/image_26 2024-09-12T06:51:23.418 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T06:51:23.422 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:23.422 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:23.422 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_27 2024-09-12T06:51:23.422 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_snapshot cluster2 mirror image_27 snap 2024-09-12T06:51:23.422 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:23.422 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:23.422 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_27 2024-09-12T06:51:23.422 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap=snap 2024-09-12T06:51:23.422 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 snap rm mirror/image_27@snap 2024-09-12T06:51:24.130 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing snap: 100% complete...done. 2024-09-12T06:51:24.185 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image_retry cluster2 mirror image_27 2024-09-12T06:51:24.186 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:24.186 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:24.186 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_27 2024-09-12T06:51:24.186 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T06:51:24.186 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:51:24.186 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image cluster2 mirror image_27 2024-09-12T06:51:24.186 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:24.186 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:24.186 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_27 2024-09-12T06:51:24.187 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_27 2024-09-12T06:51:24.196 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 rm mirror/image_27 2024-09-12T06:51:24.407 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T06:51:24.411 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:24.411 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:24.411 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_28 2024-09-12T06:51:24.411 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_snapshot cluster2 mirror image_28 snap 2024-09-12T06:51:24.411 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:24.411 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:24.411 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_28 2024-09-12T06:51:24.411 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap=snap 2024-09-12T06:51:24.411 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 snap rm mirror/image_28@snap 2024-09-12T06:51:25.041 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing snap: 100% complete...done. 2024-09-12T06:51:25.058 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image_retry cluster2 mirror image_28 2024-09-12T06:51:25.058 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:25.059 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:25.059 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_28 2024-09-12T06:51:25.059 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T06:51:25.059 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:51:25.059 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image cluster2 mirror image_28 2024-09-12T06:51:25.060 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:25.060 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:25.060 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_28 2024-09-12T06:51:25.060 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_28 2024-09-12T06:51:25.107 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 rm mirror/image_28 2024-09-12T06:51:25.304 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T06:51:25.308 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:25.308 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:25.308 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_29 2024-09-12T06:51:25.308 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_snapshot cluster2 mirror image_29 snap 2024-09-12T06:51:25.308 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:25.308 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:25.308 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_29 2024-09-12T06:51:25.308 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap=snap 2024-09-12T06:51:25.308 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 snap rm mirror/image_29@snap 2024-09-12T06:51:26.138 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing snap: 100% complete...done. 2024-09-12T06:51:26.149 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image_retry cluster2 mirror image_29 2024-09-12T06:51:26.149 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:26.149 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:26.149 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_29 2024-09-12T06:51:26.150 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T06:51:26.150 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:51:26.150 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image cluster2 mirror image_29 2024-09-12T06:51:26.150 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:26.150 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:26.151 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_29 2024-09-12T06:51:26.151 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_29 2024-09-12T06:51:26.197 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 rm mirror/image_29 2024-09-12T06:51:26.397 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T06:51:26.401 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:26.401 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:26.401 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_30 2024-09-12T06:51:26.401 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_snapshot cluster2 mirror image_30 snap 2024-09-12T06:51:26.402 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:26.402 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:26.402 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_30 2024-09-12T06:51:26.402 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap=snap 2024-09-12T06:51:26.402 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 snap rm mirror/image_30@snap 2024-09-12T06:51:27.141 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing snap: 100% complete...done. 2024-09-12T06:51:27.161 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image_retry cluster2 mirror image_30 2024-09-12T06:51:27.161 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:27.161 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:27.161 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_30 2024-09-12T06:51:27.161 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T06:51:27.161 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:51:27.162 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image cluster2 mirror image_30 2024-09-12T06:51:27.162 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:27.162 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:27.162 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_30 2024-09-12T06:51:27.162 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_30 2024-09-12T06:51:27.210 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 rm mirror/image_30 2024-09-12T06:51:27.401 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T06:51:27.405 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:27.405 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:27.405 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_31 2024-09-12T06:51:27.405 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_snapshot cluster2 mirror image_31 snap 2024-09-12T06:51:27.405 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:27.405 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:27.405 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_31 2024-09-12T06:51:27.405 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap=snap 2024-09-12T06:51:27.406 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 snap rm mirror/image_31@snap 2024-09-12T06:51:28.147 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing snap: 100% complete...done. 2024-09-12T06:51:28.160 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image_retry cluster2 mirror image_31 2024-09-12T06:51:28.160 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:28.160 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:28.160 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_31 2024-09-12T06:51:28.160 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T06:51:28.160 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:51:28.161 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image cluster2 mirror image_31 2024-09-12T06:51:28.161 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:28.161 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:28.161 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_31 2024-09-12T06:51:28.161 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_31 2024-09-12T06:51:28.209 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 rm mirror/image_31 2024-09-12T06:51:28.398 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T06:51:28.402 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:28.402 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:28.402 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_32 2024-09-12T06:51:28.402 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_snapshot cluster2 mirror image_32 snap 2024-09-12T06:51:28.402 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:28.402 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:28.402 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_32 2024-09-12T06:51:28.402 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap=snap 2024-09-12T06:51:28.402 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 snap rm mirror/image_32@snap 2024-09-12T06:51:29.153 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing snap: 100% complete...done. 2024-09-12T06:51:29.169 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image_retry cluster2 mirror image_32 2024-09-12T06:51:29.169 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:29.169 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:29.169 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_32 2024-09-12T06:51:29.169 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T06:51:29.169 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:51:29.170 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image cluster2 mirror image_32 2024-09-12T06:51:29.170 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:29.170 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:29.170 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_32 2024-09-12T06:51:29.170 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_32 2024-09-12T06:51:29.217 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 rm mirror/image_32 2024-09-12T06:51:29.353 INFO:tasks.ceph.cluster2.osd.0.smithi152.stderr:2024-09-12T06:51:29.354+0000 7fd61689f640 -1 reset not still connected to 0x55e76d1eba00 2024-09-12T06:51:29.429 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T06:51:29.433 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:29.433 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:29.433 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_33 2024-09-12T06:51:29.433 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_snapshot cluster2 mirror image_33 snap 2024-09-12T06:51:29.434 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:29.434 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:29.434 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_33 2024-09-12T06:51:29.434 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap=snap 2024-09-12T06:51:29.434 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 snap rm mirror/image_33@snap 2024-09-12T06:51:30.043 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing snap: 100% complete...done. 2024-09-12T06:51:30.054 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image_retry cluster2 mirror image_33 2024-09-12T06:51:30.055 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:30.055 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:30.055 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_33 2024-09-12T06:51:30.055 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T06:51:30.055 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:51:30.056 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image cluster2 mirror image_33 2024-09-12T06:51:30.056 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:30.056 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:30.056 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_33 2024-09-12T06:51:30.056 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_33 2024-09-12T06:51:30.103 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 rm mirror/image_33 2024-09-12T06:51:30.524 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T06:51:30.528 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:30.528 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:30.528 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_34 2024-09-12T06:51:30.528 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_snapshot cluster2 mirror image_34 snap 2024-09-12T06:51:30.529 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:30.529 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:30.529 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_34 2024-09-12T06:51:30.529 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap=snap 2024-09-12T06:51:30.529 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 snap rm mirror/image_34@snap 2024-09-12T06:51:31.156 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing snap: 100% complete...done. 2024-09-12T06:51:31.167 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image_retry cluster2 mirror image_34 2024-09-12T06:51:31.167 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:31.167 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:31.167 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_34 2024-09-12T06:51:31.167 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T06:51:31.167 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:51:31.168 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image cluster2 mirror image_34 2024-09-12T06:51:31.168 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:31.168 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:31.168 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_34 2024-09-12T06:51:31.168 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_34 2024-09-12T06:51:31.215 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 rm mirror/image_34 2024-09-12T06:51:31.415 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T06:51:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_35 2024-09-12T06:51:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_snapshot cluster2 mirror image_35 snap 2024-09-12T06:51:31.420 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:31.420 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:31.420 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_35 2024-09-12T06:51:31.420 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap=snap 2024-09-12T06:51:31.420 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 snap rm mirror/image_35@snap 2024-09-12T06:51:32.158 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing snap: 100% complete...done. 2024-09-12T06:51:32.172 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image_retry cluster2 mirror image_35 2024-09-12T06:51:32.172 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:32.172 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:32.172 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_35 2024-09-12T06:51:32.172 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T06:51:32.172 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:51:32.173 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image cluster2 mirror image_35 2024-09-12T06:51:32.173 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:32.173 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:32.173 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_35 2024-09-12T06:51:32.173 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_35 2024-09-12T06:51:32.220 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 rm mirror/image_35 2024-09-12T06:51:32.412 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T06:51:32.416 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:32.416 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:32.416 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_36 2024-09-12T06:51:32.416 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_snapshot cluster2 mirror image_36 snap 2024-09-12T06:51:32.416 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:32.416 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:32.416 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_36 2024-09-12T06:51:32.416 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap=snap 2024-09-12T06:51:32.416 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 snap rm mirror/image_36@snap 2024-09-12T06:51:33.160 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing snap: 100% complete...done. 2024-09-12T06:51:33.176 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image_retry cluster2 mirror image_36 2024-09-12T06:51:33.176 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:33.176 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:33.176 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_36 2024-09-12T06:51:33.176 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T06:51:33.176 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:51:33.177 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image cluster2 mirror image_36 2024-09-12T06:51:33.177 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:33.177 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:33.177 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_36 2024-09-12T06:51:33.177 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_36 2024-09-12T06:51:33.223 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 rm mirror/image_36 2024-09-12T06:51:33.429 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T06:51:33.433 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:33.433 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:33.433 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_37 2024-09-12T06:51:33.433 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_snapshot cluster2 mirror image_37 snap 2024-09-12T06:51:33.433 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:33.433 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:33.433 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_37 2024-09-12T06:51:33.433 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap=snap 2024-09-12T06:51:33.433 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 snap rm mirror/image_37@snap 2024-09-12T06:51:34.163 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing snap: 100% complete...done. 2024-09-12T06:51:34.182 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image_retry cluster2 mirror image_37 2024-09-12T06:51:34.182 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:34.182 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:34.182 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_37 2024-09-12T06:51:34.182 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T06:51:34.182 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:51:34.183 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image cluster2 mirror image_37 2024-09-12T06:51:34.183 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:34.183 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:34.183 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_37 2024-09-12T06:51:34.183 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_37 2024-09-12T06:51:34.228 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:51:34.229+0000 7fa325ae1640 -1 reset not still connected to 0x55f8863b4340 2024-09-12T06:51:34.228 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:51:34.229+0000 7fa325ae1640 -1 reset not still connected to 0x55f890e69a00 2024-09-12T06:51:34.229 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-12T06:51:34.230+0000 7fa325ae1640 -1 reset not still connected to 0x55f8926ce1a0 2024-09-12T06:51:34.432 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 rm mirror/image_37 2024-09-12T06:51:34.632 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T06:51:34.636 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:34.636 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:34.636 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_38 2024-09-12T06:51:34.636 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_snapshot cluster2 mirror image_38 snap 2024-09-12T06:51:34.636 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:34.636 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:34.636 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_38 2024-09-12T06:51:34.636 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap=snap 2024-09-12T06:51:34.637 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 snap rm mirror/image_38@snap 2024-09-12T06:51:34.812 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:51:34.814+0000 7fac12a52640 -1 reset not still connected to 0x55749b2be410 2024-09-12T06:51:35.039 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing snap: 100% complete...done. 2024-09-12T06:51:35.054 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image_retry cluster2 mirror image_38 2024-09-12T06:51:35.055 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:35.055 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:35.055 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_38 2024-09-12T06:51:35.055 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T06:51:35.055 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:51:35.056 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image cluster2 mirror image_38 2024-09-12T06:51:35.056 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:35.056 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:35.056 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_38 2024-09-12T06:51:35.056 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_38 2024-09-12T06:51:35.110 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 rm mirror/image_38 2024-09-12T06:51:35.300 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T06:51:35.304 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:35.304 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:35.304 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_39 2024-09-12T06:51:35.304 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_snapshot cluster2 mirror image_39 snap 2024-09-12T06:51:35.304 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:35.305 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:35.305 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_39 2024-09-12T06:51:35.305 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap=snap 2024-09-12T06:51:35.305 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 snap rm mirror/image_39@snap 2024-09-12T06:51:36.175 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing snap: 100% complete...done. 2024-09-12T06:51:36.186 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image_retry cluster2 mirror image_39 2024-09-12T06:51:36.186 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:36.187 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:36.187 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_39 2024-09-12T06:51:36.187 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T06:51:36.187 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:51:36.187 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image cluster2 mirror image_39 2024-09-12T06:51:36.187 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:36.188 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:36.188 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_39 2024-09-12T06:51:36.188 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_39 2024-09-12T06:51:36.234 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 rm mirror/image_39 2024-09-12T06:51:36.439 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T06:51:36.442 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:36.442 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:36.443 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_40 2024-09-12T06:51:36.443 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_snapshot cluster2 mirror image_40 snap 2024-09-12T06:51:36.443 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:36.443 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:36.443 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_40 2024-09-12T06:51:36.443 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap=snap 2024-09-12T06:51:36.443 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 snap rm mirror/image_40@snap 2024-09-12T06:51:37.179 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing snap: 100% complete...done. 2024-09-12T06:51:37.193 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image_retry cluster2 mirror image_40 2024-09-12T06:51:37.193 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:37.193 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:37.193 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_40 2024-09-12T06:51:37.193 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T06:51:37.193 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:51:37.194 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image cluster2 mirror image_40 2024-09-12T06:51:37.194 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:37.194 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:37.194 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_40 2024-09-12T06:51:37.194 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_40 2024-09-12T06:51:37.244 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 rm mirror/image_40 2024-09-12T06:51:37.429 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T06:51:37.433 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:37.433 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:37.433 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_41 2024-09-12T06:51:37.433 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_snapshot cluster2 mirror image_41 snap 2024-09-12T06:51:37.434 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:37.434 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:37.434 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_41 2024-09-12T06:51:37.434 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap=snap 2024-09-12T06:51:37.434 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 snap rm mirror/image_41@snap 2024-09-12T06:51:38.178 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing snap: 100% complete...done. 2024-09-12T06:51:38.190 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image_retry cluster2 mirror image_41 2024-09-12T06:51:38.190 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:38.190 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:38.190 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_41 2024-09-12T06:51:38.190 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T06:51:38.190 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:51:38.191 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image cluster2 mirror image_41 2024-09-12T06:51:38.191 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:38.191 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:38.191 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_41 2024-09-12T06:51:38.191 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_41 2024-09-12T06:51:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 rm mirror/image_41 2024-09-12T06:51:38.310 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:51:38.311+0000 7fac12a52640 -1 reset not still connected to 0x55749b2be8f0 2024-09-12T06:51:38.310 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:51:38.311+0000 7fac12a52640 -1 reset not still connected to 0x55749b2bedd0 2024-09-12T06:51:38.310 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:51:38.311+0000 7fac12a52640 -1 reset not still connected to 0x55749b2bf6c0 2024-09-12T06:51:38.313 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-12T06:51:38.313+0000 7fac12a52640 -1 reset not still connected to 0x55749c9a7860 2024-09-12T06:51:38.405 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T06:51:38.408 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:38.409 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:38.409 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_42 2024-09-12T06:51:38.409 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_snapshot cluster2 mirror image_42 snap 2024-09-12T06:51:38.409 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:38.409 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:38.409 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_42 2024-09-12T06:51:38.409 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap=snap 2024-09-12T06:51:38.409 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 snap rm mirror/image_42@snap 2024-09-12T06:51:39.195 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing snap: 100% complete...done. 2024-09-12T06:51:39.208 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image_retry cluster2 mirror image_42 2024-09-12T06:51:39.208 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:39.208 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:39.208 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_42 2024-09-12T06:51:39.208 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T06:51:39.209 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:51:39.209 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image cluster2 mirror image_42 2024-09-12T06:51:39.209 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:39.209 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:39.209 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_42 2024-09-12T06:51:39.210 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_42 2024-09-12T06:51:39.259 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 rm mirror/image_42 2024-09-12T06:51:39.442 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T06:51:39.446 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:39.446 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:39.446 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_43 2024-09-12T06:51:39.446 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_snapshot cluster2 mirror image_43 snap 2024-09-12T06:51:39.446 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:39.446 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:39.446 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_43 2024-09-12T06:51:39.446 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap=snap 2024-09-12T06:51:39.446 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 snap rm mirror/image_43@snap 2024-09-12T06:51:40.052 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing snap: 100% complete...done. 2024-09-12T06:51:40.066 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image_retry cluster2 mirror image_43 2024-09-12T06:51:40.066 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:40.066 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:40.066 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_43 2024-09-12T06:51:40.066 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T06:51:40.066 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:51:40.067 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image cluster2 mirror image_43 2024-09-12T06:51:40.067 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:40.067 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:40.067 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_43 2024-09-12T06:51:40.067 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_43 2024-09-12T06:51:40.114 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 rm mirror/image_43 2024-09-12T06:51:40.298 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T06:51:40.302 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:40.303 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:40.303 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_44 2024-09-12T06:51:40.303 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_snapshot cluster2 mirror image_44 snap 2024-09-12T06:51:40.303 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:40.303 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:40.303 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_44 2024-09-12T06:51:40.303 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap=snap 2024-09-12T06:51:40.303 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 snap rm mirror/image_44@snap 2024-09-12T06:51:41.186 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing snap: 100% complete...done. 2024-09-12T06:51:41.201 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image_retry cluster2 mirror image_44 2024-09-12T06:51:41.201 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:41.201 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:41.201 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_44 2024-09-12T06:51:41.201 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T06:51:41.202 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:51:41.202 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image cluster2 mirror image_44 2024-09-12T06:51:41.202 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:41.202 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:41.203 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_44 2024-09-12T06:51:41.203 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_44 2024-09-12T06:51:41.248 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 rm mirror/image_44 2024-09-12T06:51:41.457 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T06:51:41.462 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:41.462 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:41.462 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_45 2024-09-12T06:51:41.462 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_snapshot cluster2 mirror image_45 snap 2024-09-12T06:51:41.462 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:41.462 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:41.462 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_45 2024-09-12T06:51:41.462 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap=snap 2024-09-12T06:51:41.462 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 snap rm mirror/image_45@snap 2024-09-12T06:51:42.195 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing snap: 100% complete...done. 2024-09-12T06:51:42.208 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image_retry cluster2 mirror image_45 2024-09-12T06:51:42.208 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:42.208 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:42.208 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_45 2024-09-12T06:51:42.208 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T06:51:42.208 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:51:42.209 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image cluster2 mirror image_45 2024-09-12T06:51:42.209 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:42.209 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:42.209 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_45 2024-09-12T06:51:42.209 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_45 2024-09-12T06:51:42.255 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 rm mirror/image_45 2024-09-12T06:51:42.438 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T06:51:42.442 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:42.442 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:42.442 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_46 2024-09-12T06:51:42.442 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_snapshot cluster2 mirror image_46 snap 2024-09-12T06:51:42.443 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:42.443 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:42.443 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_46 2024-09-12T06:51:42.443 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap=snap 2024-09-12T06:51:42.443 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 snap rm mirror/image_46@snap 2024-09-12T06:51:43.193 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing snap: 100% complete...done. 2024-09-12T06:51:43.209 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image_retry cluster2 mirror image_46 2024-09-12T06:51:43.209 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:43.209 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:43.209 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_46 2024-09-12T06:51:43.209 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T06:51:43.209 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:51:43.209 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image cluster2 mirror image_46 2024-09-12T06:51:43.210 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:43.210 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:43.210 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_46 2024-09-12T06:51:43.210 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_46 2024-09-12T06:51:43.258 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 rm mirror/image_46 2024-09-12T06:51:43.651 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T06:51:43.656 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:43.656 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:43.656 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_47 2024-09-12T06:51:43.656 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_snapshot cluster2 mirror image_47 snap 2024-09-12T06:51:43.656 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:43.656 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:43.656 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_47 2024-09-12T06:51:43.656 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap=snap 2024-09-12T06:51:43.656 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 snap rm mirror/image_47@snap 2024-09-12T06:51:44.201 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing snap: 100% complete...done. 2024-09-12T06:51:44.211 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image_retry cluster2 mirror image_47 2024-09-12T06:51:44.212 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:44.212 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:44.212 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_47 2024-09-12T06:51:44.212 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T06:51:44.212 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:51:44.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image cluster2 mirror image_47 2024-09-12T06:51:44.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:44.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:44.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_47 2024-09-12T06:51:44.213 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_47 2024-09-12T06:51:44.259 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 rm mirror/image_47 2024-09-12T06:51:44.461 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T06:51:44.465 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:44.465 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:44.465 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_48 2024-09-12T06:51:44.465 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_snapshot cluster2 mirror image_48 snap 2024-09-12T06:51:44.465 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:44.465 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:44.465 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_48 2024-09-12T06:51:44.466 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap=snap 2024-09-12T06:51:44.466 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 snap rm mirror/image_48@snap 2024-09-12T06:51:45.053 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing snap: 100% complete...done. 2024-09-12T06:51:45.065 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image_retry cluster2 mirror image_48 2024-09-12T06:51:45.065 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:45.065 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:45.065 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_48 2024-09-12T06:51:45.065 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T06:51:45.065 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:51:45.066 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image cluster2 mirror image_48 2024-09-12T06:51:45.066 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:45.066 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:45.066 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_48 2024-09-12T06:51:45.066 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_48 2024-09-12T06:51:45.314 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 rm mirror/image_48 2024-09-12T06:51:45.495 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T06:51:45.499 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:45.499 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:45.499 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_49 2024-09-12T06:51:45.499 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_snapshot cluster2 mirror image_49 snap 2024-09-12T06:51:45.499 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:45.499 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:45.499 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_49 2024-09-12T06:51:45.499 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap=snap 2024-09-12T06:51:45.499 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 snap rm mirror/image_49@snap 2024-09-12T06:51:46.205 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing snap: 100% complete...done. 2024-09-12T06:51:46.218 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image_retry cluster2 mirror image_49 2024-09-12T06:51:46.218 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:46.218 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:46.218 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_49 2024-09-12T06:51:46.218 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T06:51:46.218 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:51:46.219 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image cluster2 mirror image_49 2024-09-12T06:51:46.219 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:46.219 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:46.219 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_49 2024-09-12T06:51:46.219 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_49 2024-09-12T06:51:46.266 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 rm mirror/image_49 2024-09-12T06:51:46.357 INFO:tasks.ceph.cluster2.osd.2.smithi152.stderr:2024-09-12T06:51:46.358+0000 7f4789beb640 -1 reset not still connected to 0x56163d519c70 2024-09-12T06:51:46.357 INFO:tasks.ceph.cluster2.osd.2.smithi152.stderr:2024-09-12T06:51:46.358+0000 7f4789beb640 -1 reset not still connected to 0x56163e3b2dd0 2024-09-12T06:51:46.357 INFO:tasks.ceph.cluster2.osd.2.smithi152.stderr:2024-09-12T06:51:46.358+0000 7f4789beb640 -1 reset not still connected to 0x56163e3b3d40 2024-09-12T06:51:46.357 INFO:tasks.ceph.cluster2.osd.2.smithi152.stderr:2024-09-12T06:51:46.358+0000 7f4789beb640 -1 reset not still connected to 0x56163e3b3ee0 2024-09-12T06:51:46.454 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T06:51:46.458 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:46.458 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:46.458 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_50 2024-09-12T06:51:46.458 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_snapshot cluster2 mirror image_50 snap 2024-09-12T06:51:46.458 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:46.458 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:46.458 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_50 2024-09-12T06:51:46.459 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local snap=snap 2024-09-12T06:51:46.459 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 snap rm mirror/image_50@snap 2024-09-12T06:51:47.205 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing snap: 100% complete...done. 2024-09-12T06:51:47.216 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image_retry cluster2 mirror image_50 2024-09-12T06:51:47.217 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:47.217 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:47.217 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_50 2024-09-12T06:51:47.217 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T06:51:47.217 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:51:47.218 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image cluster2 mirror image_50 2024-09-12T06:51:47.218 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:47.218 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:47.218 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_50 2024-09-12T06:51:47.218 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_50 2024-09-12T06:51:47.264 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 rm mirror/image_50 2024-09-12T06:51:47.458 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T06:51:47.461 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:47.462 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ testlog 'TEST: image deletions should propagate' 2024-09-12T06:51:47.462 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ tee -a /tmp/tmp.GzjHZGs9Z0/rbd-mirror.test.log 2024-09-12T06:51:47.462 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ date '+%F %T' 2024-09-12T06:51:47.463 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo 2024-09-12 06:51:47 TEST: image deletions should propagate 2024-09-12T06:51:47.464 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:2024-09-12 06:51:47 TEST: image deletions should propagate 2024-09-12T06:51:47.464 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_pool_images cluster1 mirror 0 2024-09-12T06:51:47.464 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:47.464 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:47.464 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_count=0 2024-09-12T06:51:47.464 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:51:47.464 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local count 2024-09-12T06:51:47.464 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local last_count=0 2024-09-12T06:51:47.464 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ true 2024-09-12T06:51:47.464 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 1 40 2024-09-12T06:51:47.465 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in `seq 1 40` 2024-09-12T06:51:47.465 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 1 -ne 1 2024-09-12T06:51:47.465 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-09-12T06:51:47.465 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep 'images: ' 2024-09-12T06:51:47.465 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut '-d ' -f 2 2024-09-12T06:51:47.501 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: failed to query services: (13) Permission denied 2024-09-12T06:51:47.505 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ count=0 2024-09-12T06:51:47.505 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 0 = 0 2024-09-12T06:51:47.505 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:47.505 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_pool_healthy cluster1 mirror 0 2024-09-12T06:51:47.505 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:47.505 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:47.505 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:51:47.505 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state 2024-09-12T06:51:47.506 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 1 40 2024-09-12T06:51:47.506 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in `seq 1 40` 2024-09-12T06:51:47.506 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 1 -ne 1 2024-09-12T06:51:47.506 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-09-12T06:51:47.507 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep 'image health:' 2024-09-12T06:51:47.507 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cut '-d ' -f 3 2024-09-12T06:51:47.543 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: failed to query services: (13) Permission denied 2024-09-12T06:51:47.546 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ state=OK 2024-09-12T06:51:47.546 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test OK = ERROR 2024-09-12T06:51:47.546 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test OK = OK 2024-09-12T06:51:47.546 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:47.546 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 1 50 2024-09-12T06:51:47.548 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:47.548 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_1 2024-09-12T06:51:47.548 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_present cluster1 mirror image_1 deleted 2024-09-12T06:51:47.548 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:47.548 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:47.548 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_1 2024-09-12T06:51:47.548 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=deleted 2024-09-12T06:51:47.548 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:47.548 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:51:47.548 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:47.548 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_1 2024-09-12T06:51:47.548 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:47.549 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:47.549 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_1 2024-09-12T06:51:47.549 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-12T06:51:47.549 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:47.581 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-09-12T06:51:47.585 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image_id= 2024-09-12T06:51:47.585 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T06:51:47.585 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.1 2024-09-12T06:51:47.686 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_present cluster1 mirror image_1 deleted '' 2024-09-12T06:51:47.687 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:47.687 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:47.687 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_1 2024-09-12T06:51:47.687 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=deleted 2024-09-12T06:51:47.687 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:47.687 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=deleted 2024-09-12T06:51:47.687 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_image_id 2024-09-12T06:51:47.687 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_1 2024-09-12T06:51:47.687 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:47.687 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:47.687 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_1 2024-09-12T06:51:47.687 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-12T06:51:47.688 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:47.721 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-09-12T06:51:47.725 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_image_id= 2024-09-12T06:51:47.725 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:47.726 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test deleted = deleted 2024-09-12T06:51:47.726 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:47.726 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:47.726 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_2 2024-09-12T06:51:47.726 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_present cluster1 mirror image_2 deleted 2024-09-12T06:51:47.726 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:47.726 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:47.726 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_2 2024-09-12T06:51:47.726 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=deleted 2024-09-12T06:51:47.726 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:47.726 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:51:47.726 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:47.726 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_2 2024-09-12T06:51:47.727 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:47.727 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:47.727 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_2 2024-09-12T06:51:47.727 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-12T06:51:47.727 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:47.760 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-09-12T06:51:47.763 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image_id= 2024-09-12T06:51:47.763 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T06:51:47.763 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.1 2024-09-12T06:51:47.865 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_present cluster1 mirror image_2 deleted '' 2024-09-12T06:51:47.865 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:47.865 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:47.865 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_2 2024-09-12T06:51:47.865 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=deleted 2024-09-12T06:51:47.866 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:47.866 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=deleted 2024-09-12T06:51:47.866 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_image_id 2024-09-12T06:51:47.866 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_2 2024-09-12T06:51:47.866 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:47.866 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:47.866 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_2 2024-09-12T06:51:47.866 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-12T06:51:47.866 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:47.900 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-09-12T06:51:47.904 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_image_id= 2024-09-12T06:51:47.904 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:47.905 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test deleted = deleted 2024-09-12T06:51:47.905 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:47.905 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:47.905 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_3 2024-09-12T06:51:47.905 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_present cluster1 mirror image_3 deleted 2024-09-12T06:51:47.905 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:47.905 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:47.905 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_3 2024-09-12T06:51:47.905 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=deleted 2024-09-12T06:51:47.905 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:47.905 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:51:47.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:47.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_3 2024-09-12T06:51:47.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:47.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:47.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_3 2024-09-12T06:51:47.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-12T06:51:47.906 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:47.939 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-09-12T06:51:47.942 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image_id= 2024-09-12T06:51:47.942 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T06:51:47.942 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.1 2024-09-12T06:51:48.044 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_present cluster1 mirror image_3 deleted '' 2024-09-12T06:51:48.045 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:48.045 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:48.045 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_3 2024-09-12T06:51:48.045 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=deleted 2024-09-12T06:51:48.045 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:48.045 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=deleted 2024-09-12T06:51:48.045 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_image_id 2024-09-12T06:51:48.045 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_3 2024-09-12T06:51:48.045 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:48.045 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:48.045 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_3 2024-09-12T06:51:48.045 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-12T06:51:48.046 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:48.079 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-09-12T06:51:48.084 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_image_id= 2024-09-12T06:51:48.084 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:48.084 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test deleted = deleted 2024-09-12T06:51:48.084 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:48.084 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:48.084 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_4 2024-09-12T06:51:48.085 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_present cluster1 mirror image_4 deleted 2024-09-12T06:51:48.085 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:48.085 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:48.085 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_4 2024-09-12T06:51:48.085 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=deleted 2024-09-12T06:51:48.085 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:48.085 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:51:48.085 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:48.085 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_4 2024-09-12T06:51:48.085 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:48.085 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:48.085 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_4 2024-09-12T06:51:48.085 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-12T06:51:48.086 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:48.118 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-09-12T06:51:48.121 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image_id= 2024-09-12T06:51:48.121 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T06:51:48.121 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.1 2024-09-12T06:51:48.224 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_present cluster1 mirror image_4 deleted '' 2024-09-12T06:51:48.224 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:48.224 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:48.224 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_4 2024-09-12T06:51:48.224 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=deleted 2024-09-12T06:51:48.224 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:48.224 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=deleted 2024-09-12T06:51:48.224 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_image_id 2024-09-12T06:51:48.224 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_4 2024-09-12T06:51:48.224 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:48.225 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:48.225 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_4 2024-09-12T06:51:48.225 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-12T06:51:48.225 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:48.261 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-09-12T06:51:48.266 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_image_id= 2024-09-12T06:51:48.266 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:48.266 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test deleted = deleted 2024-09-12T06:51:48.266 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:48.266 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:48.266 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_5 2024-09-12T06:51:48.266 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_present cluster1 mirror image_5 deleted 2024-09-12T06:51:48.266 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:48.266 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:48.266 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_5 2024-09-12T06:51:48.266 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=deleted 2024-09-12T06:51:48.267 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:48.267 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:51:48.267 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:48.267 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_5 2024-09-12T06:51:48.267 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:48.267 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:48.267 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_5 2024-09-12T06:51:48.267 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-12T06:51:48.267 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:48.319 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-09-12T06:51:48.323 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image_id= 2024-09-12T06:51:48.323 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T06:51:48.323 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.1 2024-09-12T06:51:48.425 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_present cluster1 mirror image_5 deleted '' 2024-09-12T06:51:48.425 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:48.425 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:48.425 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_5 2024-09-12T06:51:48.425 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=deleted 2024-09-12T06:51:48.425 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:48.425 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=deleted 2024-09-12T06:51:48.425 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_image_id 2024-09-12T06:51:48.425 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_5 2024-09-12T06:51:48.425 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:48.425 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:48.426 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_5 2024-09-12T06:51:48.426 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-12T06:51:48.426 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:48.460 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-09-12T06:51:48.465 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_image_id= 2024-09-12T06:51:48.465 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:48.465 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test deleted = deleted 2024-09-12T06:51:48.465 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:48.465 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:48.465 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_6 2024-09-12T06:51:48.465 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_present cluster1 mirror image_6 deleted 2024-09-12T06:51:48.465 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:48.465 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:48.465 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_6 2024-09-12T06:51:48.466 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=deleted 2024-09-12T06:51:48.466 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:48.466 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:51:48.466 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:48.466 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_6 2024-09-12T06:51:48.466 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:48.466 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:48.466 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_6 2024-09-12T06:51:48.466 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-12T06:51:48.466 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:48.500 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-09-12T06:51:48.503 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image_id= 2024-09-12T06:51:48.503 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T06:51:48.503 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.1 2024-09-12T06:51:48.604 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_present cluster1 mirror image_6 deleted '' 2024-09-12T06:51:48.605 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:48.605 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:48.605 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_6 2024-09-12T06:51:48.605 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=deleted 2024-09-12T06:51:48.605 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:48.605 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=deleted 2024-09-12T06:51:48.605 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_image_id 2024-09-12T06:51:48.605 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_6 2024-09-12T06:51:48.605 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:48.605 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:48.605 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_6 2024-09-12T06:51:48.605 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-12T06:51:48.605 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:48.641 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-09-12T06:51:48.645 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_image_id= 2024-09-12T06:51:48.645 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:48.645 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test deleted = deleted 2024-09-12T06:51:48.645 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:48.646 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:48.646 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_7 2024-09-12T06:51:48.646 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_present cluster1 mirror image_7 deleted 2024-09-12T06:51:48.646 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:48.646 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:48.646 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_7 2024-09-12T06:51:48.646 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=deleted 2024-09-12T06:51:48.646 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:48.646 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:51:48.646 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:48.646 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_7 2024-09-12T06:51:48.646 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:48.646 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:48.646 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_7 2024-09-12T06:51:48.647 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-12T06:51:48.647 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:48.680 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-09-12T06:51:48.683 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image_id= 2024-09-12T06:51:48.683 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T06:51:48.683 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.1 2024-09-12T06:51:48.785 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_present cluster1 mirror image_7 deleted '' 2024-09-12T06:51:48.785 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:48.785 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:48.785 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_7 2024-09-12T06:51:48.785 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=deleted 2024-09-12T06:51:48.785 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:48.785 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=deleted 2024-09-12T06:51:48.785 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_image_id 2024-09-12T06:51:48.785 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_7 2024-09-12T06:51:48.785 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:48.785 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:48.785 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_7 2024-09-12T06:51:48.786 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-12T06:51:48.786 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:48.822 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-09-12T06:51:48.826 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_image_id= 2024-09-12T06:51:48.826 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:48.826 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test deleted = deleted 2024-09-12T06:51:48.826 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:48.826 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:48.826 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_8 2024-09-12T06:51:48.826 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_present cluster1 mirror image_8 deleted 2024-09-12T06:51:48.826 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:48.826 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:48.827 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_8 2024-09-12T06:51:48.827 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=deleted 2024-09-12T06:51:48.827 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:48.827 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:51:48.827 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:48.827 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_8 2024-09-12T06:51:48.827 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:48.827 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:48.827 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_8 2024-09-12T06:51:48.827 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-12T06:51:48.827 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:48.861 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-09-12T06:51:48.864 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image_id= 2024-09-12T06:51:48.865 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T06:51:48.865 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.1 2024-09-12T06:51:48.966 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_present cluster1 mirror image_8 deleted '' 2024-09-12T06:51:48.967 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:48.967 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:48.967 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_8 2024-09-12T06:51:48.967 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=deleted 2024-09-12T06:51:48.967 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:48.967 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=deleted 2024-09-12T06:51:48.967 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_image_id 2024-09-12T06:51:48.967 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_8 2024-09-12T06:51:48.967 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:48.967 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:48.967 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_8 2024-09-12T06:51:48.967 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-12T06:51:48.967 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-09-12T06:51:49.006 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_image_id= 2024-09-12T06:51:49.006 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:49.007 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test deleted = deleted 2024-09-12T06:51:49.007 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:49.007 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:49.007 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_9 2024-09-12T06:51:49.007 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_present cluster1 mirror image_9 deleted 2024-09-12T06:51:49.007 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:49.007 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:49.007 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_9 2024-09-12T06:51:49.007 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=deleted 2024-09-12T06:51:49.007 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:49.007 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:51:49.007 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:49.007 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_9 2024-09-12T06:51:49.007 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:49.008 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:49.008 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_9 2024-09-12T06:51:49.008 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-12T06:51:49.008 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:49.242 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-09-12T06:51:49.246 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image_id= 2024-09-12T06:51:49.246 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T06:51:49.246 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.1 2024-09-12T06:51:49.349 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_present cluster1 mirror image_9 deleted '' 2024-09-12T06:51:49.349 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:49.349 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:49.349 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_9 2024-09-12T06:51:49.349 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=deleted 2024-09-12T06:51:49.349 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:49.349 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=deleted 2024-09-12T06:51:49.349 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_image_id 2024-09-12T06:51:49.350 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_9 2024-09-12T06:51:49.350 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:49.350 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:49.350 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_9 2024-09-12T06:51:49.350 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-12T06:51:49.350 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-09-12T06:51:49.394 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_image_id= 2024-09-12T06:51:49.394 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:49.395 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test deleted = deleted 2024-09-12T06:51:49.395 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:49.395 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:49.395 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_10 2024-09-12T06:51:49.395 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_present cluster1 mirror image_10 deleted 2024-09-12T06:51:49.396 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:49.396 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:49.396 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_10 2024-09-12T06:51:49.396 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=deleted 2024-09-12T06:51:49.396 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:49.396 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:51:49.397 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:49.397 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_10 2024-09-12T06:51:49.397 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:49.397 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:49.397 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_10 2024-09-12T06:51:49.398 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-12T06:51:49.398 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:49.428 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-09-12T06:51:49.432 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image_id= 2024-09-12T06:51:49.432 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T06:51:49.432 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.1 2024-09-12T06:51:49.534 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_present cluster1 mirror image_10 deleted '' 2024-09-12T06:51:49.534 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:49.534 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:49.534 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_10 2024-09-12T06:51:49.535 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=deleted 2024-09-12T06:51:49.535 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:49.535 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=deleted 2024-09-12T06:51:49.535 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_image_id 2024-09-12T06:51:49.535 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_10 2024-09-12T06:51:49.535 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:49.535 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:49.535 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_10 2024-09-12T06:51:49.535 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-12T06:51:49.535 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:49.570 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-09-12T06:51:49.575 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_image_id= 2024-09-12T06:51:49.575 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:49.575 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test deleted = deleted 2024-09-12T06:51:49.575 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:49.575 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:49.575 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_11 2024-09-12T06:51:49.576 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_present cluster1 mirror image_11 deleted 2024-09-12T06:51:49.576 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:49.576 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:49.576 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_11 2024-09-12T06:51:49.576 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=deleted 2024-09-12T06:51:49.576 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:49.576 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:51:49.576 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:49.576 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_11 2024-09-12T06:51:49.576 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:49.576 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:49.576 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_11 2024-09-12T06:51:49.576 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_11 2024-09-12T06:51:49.577 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:49.611 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_11: (2) No such file or directory 2024-09-12T06:51:49.614 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image_id= 2024-09-12T06:51:49.614 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T06:51:49.614 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.1 2024-09-12T06:51:49.716 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_present cluster1 mirror image_11 deleted '' 2024-09-12T06:51:49.717 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:49.717 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:49.717 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_11 2024-09-12T06:51:49.717 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=deleted 2024-09-12T06:51:49.717 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:49.717 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=deleted 2024-09-12T06:51:49.717 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_image_id 2024-09-12T06:51:49.717 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_11 2024-09-12T06:51:49.717 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:49.717 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:49.717 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_11 2024-09-12T06:51:49.717 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_11 2024-09-12T06:51:49.717 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:49.752 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_11: (2) No such file or directory 2024-09-12T06:51:49.757 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_image_id= 2024-09-12T06:51:49.757 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:49.757 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test deleted = deleted 2024-09-12T06:51:49.757 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:49.757 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:49.757 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_12 2024-09-12T06:51:49.757 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_present cluster1 mirror image_12 deleted 2024-09-12T06:51:49.757 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:49.757 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:49.757 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_12 2024-09-12T06:51:49.757 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=deleted 2024-09-12T06:51:49.758 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:49.758 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:51:49.758 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:49.758 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_12 2024-09-12T06:51:49.758 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:49.758 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:49.758 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_12 2024-09-12T06:51:49.758 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_12 2024-09-12T06:51:49.758 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:49.791 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_12: (2) No such file or directory 2024-09-12T06:51:49.794 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image_id= 2024-09-12T06:51:49.794 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T06:51:49.794 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.1 2024-09-12T06:51:49.896 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_present cluster1 mirror image_12 deleted '' 2024-09-12T06:51:49.896 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:49.897 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:49.897 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_12 2024-09-12T06:51:49.897 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=deleted 2024-09-12T06:51:49.897 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:49.897 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=deleted 2024-09-12T06:51:49.897 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_image_id 2024-09-12T06:51:49.897 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_12 2024-09-12T06:51:49.897 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:49.897 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:49.897 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_12 2024-09-12T06:51:49.897 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_12 2024-09-12T06:51:49.897 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:49.931 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_12: (2) No such file or directory 2024-09-12T06:51:49.936 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_image_id= 2024-09-12T06:51:49.936 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:49.936 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test deleted = deleted 2024-09-12T06:51:49.936 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:49.936 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:49.936 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_13 2024-09-12T06:51:49.936 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_present cluster1 mirror image_13 deleted 2024-09-12T06:51:49.936 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:49.937 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:49.937 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_13 2024-09-12T06:51:49.937 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=deleted 2024-09-12T06:51:49.937 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:49.937 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:51:49.937 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:49.937 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_13 2024-09-12T06:51:49.937 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:49.937 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:49.937 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_13 2024-09-12T06:51:49.937 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_13 2024-09-12T06:51:49.937 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:49.971 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_13: (2) No such file or directory 2024-09-12T06:51:49.974 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image_id= 2024-09-12T06:51:49.974 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T06:51:49.975 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.1 2024-09-12T06:51:50.076 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_present cluster1 mirror image_13 deleted '' 2024-09-12T06:51:50.077 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:50.077 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:50.077 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_13 2024-09-12T06:51:50.077 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=deleted 2024-09-12T06:51:50.077 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:50.077 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=deleted 2024-09-12T06:51:50.077 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_image_id 2024-09-12T06:51:50.077 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_13 2024-09-12T06:51:50.077 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:50.077 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:50.078 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_13 2024-09-12T06:51:50.078 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_13 2024-09-12T06:51:50.078 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:50.113 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_13: (2) No such file or directory 2024-09-12T06:51:50.117 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_image_id= 2024-09-12T06:51:50.117 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:50.117 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test deleted = deleted 2024-09-12T06:51:50.117 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:50.118 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:50.118 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_14 2024-09-12T06:51:50.118 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_present cluster1 mirror image_14 deleted 2024-09-12T06:51:50.118 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:50.118 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:50.118 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_14 2024-09-12T06:51:50.118 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=deleted 2024-09-12T06:51:50.118 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:50.119 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:51:50.119 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:50.119 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_14 2024-09-12T06:51:50.119 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:50.119 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:50.120 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_14 2024-09-12T06:51:50.120 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_14 2024-09-12T06:51:50.120 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:50.153 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_14: (2) No such file or directory 2024-09-12T06:51:50.156 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image_id= 2024-09-12T06:51:50.156 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T06:51:50.156 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.1 2024-09-12T06:51:50.258 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_present cluster1 mirror image_14 deleted '' 2024-09-12T06:51:50.259 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:50.259 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:50.259 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_14 2024-09-12T06:51:50.259 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=deleted 2024-09-12T06:51:50.259 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:50.259 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=deleted 2024-09-12T06:51:50.259 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_image_id 2024-09-12T06:51:50.259 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_14 2024-09-12T06:51:50.259 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:50.259 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:50.259 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_14 2024-09-12T06:51:50.260 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_14 2024-09-12T06:51:50.260 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:50.295 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_14: (2) No such file or directory 2024-09-12T06:51:50.303 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_image_id= 2024-09-12T06:51:50.303 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:50.303 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test deleted = deleted 2024-09-12T06:51:50.303 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:50.303 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:50.303 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_15 2024-09-12T06:51:50.303 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_present cluster1 mirror image_15 deleted 2024-09-12T06:51:50.303 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:50.304 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:50.304 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_15 2024-09-12T06:51:50.304 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=deleted 2024-09-12T06:51:50.304 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:50.304 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:51:50.304 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:50.304 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_15 2024-09-12T06:51:50.304 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:50.304 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:50.304 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_15 2024-09-12T06:51:50.304 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_15 2024-09-12T06:51:50.304 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:50.335 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_15: (2) No such file or directory 2024-09-12T06:51:50.338 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image_id= 2024-09-12T06:51:50.338 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T06:51:50.339 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.1 2024-09-12T06:51:50.440 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_present cluster1 mirror image_15 deleted '' 2024-09-12T06:51:50.440 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:50.440 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:50.440 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_15 2024-09-12T06:51:50.440 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=deleted 2024-09-12T06:51:50.440 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:50.441 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=deleted 2024-09-12T06:51:50.441 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_image_id 2024-09-12T06:51:50.441 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_15 2024-09-12T06:51:50.441 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:50.441 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:50.441 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_15 2024-09-12T06:51:50.441 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_15 2024-09-12T06:51:50.441 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:50.476 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_15: (2) No such file or directory 2024-09-12T06:51:50.480 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_image_id= 2024-09-12T06:51:50.480 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:50.480 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test deleted = deleted 2024-09-12T06:51:50.480 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:50.480 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:50.481 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_16 2024-09-12T06:51:50.481 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_present cluster1 mirror image_16 deleted 2024-09-12T06:51:50.481 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:50.481 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:50.481 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_16 2024-09-12T06:51:50.481 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=deleted 2024-09-12T06:51:50.481 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:50.481 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:51:50.481 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:50.481 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_16 2024-09-12T06:51:50.481 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:50.481 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:50.481 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_16 2024-09-12T06:51:50.481 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_16 2024-09-12T06:51:50.482 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:50.514 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_16: (2) No such file or directory 2024-09-12T06:51:50.517 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image_id= 2024-09-12T06:51:50.517 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T06:51:50.518 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.1 2024-09-12T06:51:50.619 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_present cluster1 mirror image_16 deleted '' 2024-09-12T06:51:50.620 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:50.620 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:50.620 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_16 2024-09-12T06:51:50.620 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=deleted 2024-09-12T06:51:50.620 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:50.620 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=deleted 2024-09-12T06:51:50.620 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_image_id 2024-09-12T06:51:50.620 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_16 2024-09-12T06:51:50.620 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:50.620 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:50.620 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_16 2024-09-12T06:51:50.620 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_16 2024-09-12T06:51:50.621 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:50.656 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_16: (2) No such file or directory 2024-09-12T06:51:50.661 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_image_id= 2024-09-12T06:51:50.661 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:50.661 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test deleted = deleted 2024-09-12T06:51:50.661 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:50.661 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:50.661 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_17 2024-09-12T06:51:50.661 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_present cluster1 mirror image_17 deleted 2024-09-12T06:51:50.662 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:50.662 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:50.662 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_17 2024-09-12T06:51:50.662 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=deleted 2024-09-12T06:51:50.662 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:50.662 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:51:50.662 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:50.662 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_17 2024-09-12T06:51:50.662 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:50.662 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:50.662 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_17 2024-09-12T06:51:50.662 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_17 2024-09-12T06:51:50.663 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:50.696 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_17: (2) No such file or directory 2024-09-12T06:51:50.700 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image_id= 2024-09-12T06:51:50.700 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T06:51:50.700 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.1 2024-09-12T06:51:50.802 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_present cluster1 mirror image_17 deleted '' 2024-09-12T06:51:50.802 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:50.802 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:50.802 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_17 2024-09-12T06:51:50.803 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=deleted 2024-09-12T06:51:50.803 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:50.803 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=deleted 2024-09-12T06:51:50.803 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_image_id 2024-09-12T06:51:50.803 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_17 2024-09-12T06:51:50.803 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:50.803 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:50.804 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_17 2024-09-12T06:51:50.804 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_17 2024-09-12T06:51:50.804 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:50.837 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_17: (2) No such file or directory 2024-09-12T06:51:50.842 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_image_id= 2024-09-12T06:51:50.843 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:50.843 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test deleted = deleted 2024-09-12T06:51:50.843 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:50.843 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:50.843 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_18 2024-09-12T06:51:50.843 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_present cluster1 mirror image_18 deleted 2024-09-12T06:51:50.843 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:50.843 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:50.843 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_18 2024-09-12T06:51:50.843 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=deleted 2024-09-12T06:51:50.843 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:50.843 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:51:50.844 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:50.844 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_18 2024-09-12T06:51:50.844 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:50.844 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:50.844 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_18 2024-09-12T06:51:50.844 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_18 2024-09-12T06:51:50.844 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:50.877 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_18: (2) No such file or directory 2024-09-12T06:51:50.880 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image_id= 2024-09-12T06:51:50.881 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T06:51:50.881 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.1 2024-09-12T06:51:50.982 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_present cluster1 mirror image_18 deleted '' 2024-09-12T06:51:50.983 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:50.983 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:50.983 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_18 2024-09-12T06:51:50.983 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=deleted 2024-09-12T06:51:50.983 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:50.983 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=deleted 2024-09-12T06:51:50.983 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_image_id 2024-09-12T06:51:50.983 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_18 2024-09-12T06:51:50.984 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:50.984 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:50.984 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_18 2024-09-12T06:51:50.984 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_18 2024-09-12T06:51:50.984 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:51.018 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_18: (2) No such file or directory 2024-09-12T06:51:51.023 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_image_id= 2024-09-12T06:51:51.023 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:51.023 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test deleted = deleted 2024-09-12T06:51:51.023 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:51.023 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:51.023 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_19 2024-09-12T06:51:51.023 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_present cluster1 mirror image_19 deleted 2024-09-12T06:51:51.023 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:51.023 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:51.023 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_19 2024-09-12T06:51:51.024 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=deleted 2024-09-12T06:51:51.024 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:51.024 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:51:51.024 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:51.024 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_19 2024-09-12T06:51:51.024 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:51.024 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:51.024 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_19 2024-09-12T06:51:51.024 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_19 2024-09-12T06:51:51.024 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:51.058 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_19: (2) No such file or directory 2024-09-12T06:51:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image_id= 2024-09-12T06:51:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T06:51:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.1 2024-09-12T06:51:51.163 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_present cluster1 mirror image_19 deleted '' 2024-09-12T06:51:51.163 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:51.163 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:51.163 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_19 2024-09-12T06:51:51.163 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=deleted 2024-09-12T06:51:51.163 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:51.163 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=deleted 2024-09-12T06:51:51.163 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_image_id 2024-09-12T06:51:51.164 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_19 2024-09-12T06:51:51.164 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:51.164 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:51.164 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_19 2024-09-12T06:51:51.164 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_19 2024-09-12T06:51:51.164 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:51.201 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_19: (2) No such file or directory 2024-09-12T06:51:51.205 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_image_id= 2024-09-12T06:51:51.206 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:51.206 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test deleted = deleted 2024-09-12T06:51:51.206 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:51.206 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:51.206 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_20 2024-09-12T06:51:51.206 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_present cluster1 mirror image_20 deleted 2024-09-12T06:51:51.206 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:51.206 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:51.206 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_20 2024-09-12T06:51:51.206 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=deleted 2024-09-12T06:51:51.206 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:51.206 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:51:51.206 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:51.207 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_20 2024-09-12T06:51:51.207 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:51.207 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:51.207 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_20 2024-09-12T06:51:51.207 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_20 2024-09-12T06:51:51.207 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:51.240 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_20: (2) No such file or directory 2024-09-12T06:51:51.243 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image_id= 2024-09-12T06:51:51.243 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T06:51:51.243 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.1 2024-09-12T06:51:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_present cluster1 mirror image_20 deleted '' 2024-09-12T06:51:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_20 2024-09-12T06:51:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=deleted 2024-09-12T06:51:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=deleted 2024-09-12T06:51:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_image_id 2024-09-12T06:51:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_20 2024-09-12T06:51:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_20 2024-09-12T06:51:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_20 2024-09-12T06:51:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:51.381 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_20: (2) No such file or directory 2024-09-12T06:51:51.386 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_image_id= 2024-09-12T06:51:51.386 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:51.386 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test deleted = deleted 2024-09-12T06:51:51.386 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:51.386 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:51.386 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_21 2024-09-12T06:51:51.386 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_present cluster1 mirror image_21 deleted 2024-09-12T06:51:51.386 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:51.386 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:51.386 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_21 2024-09-12T06:51:51.386 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=deleted 2024-09-12T06:51:51.386 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:51.386 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:51:51.387 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:51.387 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_21 2024-09-12T06:51:51.387 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:51.387 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:51.387 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_21 2024-09-12T06:51:51.387 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_21 2024-09-12T06:51:51.387 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:51.420 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_21: (2) No such file or directory 2024-09-12T06:51:51.423 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image_id= 2024-09-12T06:51:51.423 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T06:51:51.423 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.1 2024-09-12T06:51:51.525 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_present cluster1 mirror image_21 deleted '' 2024-09-12T06:51:51.525 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:51.525 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:51.525 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_21 2024-09-12T06:51:51.526 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=deleted 2024-09-12T06:51:51.526 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:51.526 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=deleted 2024-09-12T06:51:51.526 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_image_id 2024-09-12T06:51:51.526 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_21 2024-09-12T06:51:51.526 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:51.526 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:51.526 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_21 2024-09-12T06:51:51.526 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_21 2024-09-12T06:51:51.526 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:51.561 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_21: (2) No such file or directory 2024-09-12T06:51:51.566 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_image_id= 2024-09-12T06:51:51.566 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:51.566 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test deleted = deleted 2024-09-12T06:51:51.566 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:51.566 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:51.566 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_22 2024-09-12T06:51:51.566 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_present cluster1 mirror image_22 deleted 2024-09-12T06:51:51.567 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:51.567 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:51.567 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_22 2024-09-12T06:51:51.567 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=deleted 2024-09-12T06:51:51.567 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:51.567 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:51:51.567 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:51.567 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_22 2024-09-12T06:51:51.567 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:51.567 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:51.567 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_22 2024-09-12T06:51:51.567 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_22 2024-09-12T06:51:51.567 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:51.602 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_22: (2) No such file or directory 2024-09-12T06:51:51.605 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image_id= 2024-09-12T06:51:51.605 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T06:51:51.605 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.1 2024-09-12T06:51:51.707 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_present cluster1 mirror image_22 deleted '' 2024-09-12T06:51:51.707 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:51.707 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:51.707 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_22 2024-09-12T06:51:51.708 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=deleted 2024-09-12T06:51:51.708 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:51.708 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=deleted 2024-09-12T06:51:51.708 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_image_id 2024-09-12T06:51:51.708 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_22 2024-09-12T06:51:51.708 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:51.708 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:51.708 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_22 2024-09-12T06:51:51.708 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_22 2024-09-12T06:51:51.708 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:51.947 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_22: (2) No such file or directory 2024-09-12T06:51:51.952 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_image_id= 2024-09-12T06:51:51.952 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:51.952 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test deleted = deleted 2024-09-12T06:51:51.952 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:51.953 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:51.953 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_23 2024-09-12T06:51:51.953 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_present cluster1 mirror image_23 deleted 2024-09-12T06:51:51.953 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:51.953 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:51.953 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_23 2024-09-12T06:51:51.953 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=deleted 2024-09-12T06:51:51.953 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:51.953 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:51:51.953 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:51.953 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_23 2024-09-12T06:51:51.953 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:51.953 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:51.954 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_23 2024-09-12T06:51:51.954 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_23 2024-09-12T06:51:51.954 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:51.990 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_23: (2) No such file or directory 2024-09-12T06:51:51.993 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image_id= 2024-09-12T06:51:51.993 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T06:51:51.993 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.1 2024-09-12T06:51:52.095 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_present cluster1 mirror image_23 deleted '' 2024-09-12T06:51:52.095 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:52.095 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:52.095 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_23 2024-09-12T06:51:52.095 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=deleted 2024-09-12T06:51:52.095 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:52.095 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=deleted 2024-09-12T06:51:52.095 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_image_id 2024-09-12T06:51:52.096 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_23 2024-09-12T06:51:52.096 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:52.096 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:52.096 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_23 2024-09-12T06:51:52.096 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_23 2024-09-12T06:51:52.096 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:52.131 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_23: (2) No such file or directory 2024-09-12T06:51:52.135 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_image_id= 2024-09-12T06:51:52.135 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:52.135 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test deleted = deleted 2024-09-12T06:51:52.135 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:52.135 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:52.136 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_24 2024-09-12T06:51:52.136 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_present cluster1 mirror image_24 deleted 2024-09-12T06:51:52.136 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:52.136 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:52.136 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_24 2024-09-12T06:51:52.136 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=deleted 2024-09-12T06:51:52.136 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:52.136 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:51:52.136 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:52.136 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_24 2024-09-12T06:51:52.136 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:52.136 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:52.136 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_24 2024-09-12T06:51:52.137 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_24 2024-09-12T06:51:52.137 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:52.171 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_24: (2) No such file or directory 2024-09-12T06:51:52.174 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image_id= 2024-09-12T06:51:52.174 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T06:51:52.174 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.1 2024-09-12T06:51:52.276 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_present cluster1 mirror image_24 deleted '' 2024-09-12T06:51:52.276 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:52.276 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:52.276 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_24 2024-09-12T06:51:52.276 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=deleted 2024-09-12T06:51:52.276 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:52.276 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=deleted 2024-09-12T06:51:52.276 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_image_id 2024-09-12T06:51:52.276 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_24 2024-09-12T06:51:52.277 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:52.277 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:52.277 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_24 2024-09-12T06:51:52.277 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_24 2024-09-12T06:51:52.277 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:52.313 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_24: (2) No such file or directory 2024-09-12T06:51:52.317 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_image_id= 2024-09-12T06:51:52.317 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:52.317 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test deleted = deleted 2024-09-12T06:51:52.317 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:52.317 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:52.317 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_25 2024-09-12T06:51:52.318 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_present cluster1 mirror image_25 deleted 2024-09-12T06:51:52.318 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:52.318 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:52.318 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_25 2024-09-12T06:51:52.318 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=deleted 2024-09-12T06:51:52.318 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:52.318 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:51:52.318 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:52.318 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_25 2024-09-12T06:51:52.318 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:52.318 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:52.318 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_25 2024-09-12T06:51:52.318 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_25 2024-09-12T06:51:52.318 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:52.352 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_25: (2) No such file or directory 2024-09-12T06:51:52.355 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image_id= 2024-09-12T06:51:52.355 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T06:51:52.355 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.1 2024-09-12T06:51:52.457 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_present cluster1 mirror image_25 deleted '' 2024-09-12T06:51:52.457 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:52.457 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:52.457 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_25 2024-09-12T06:51:52.457 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=deleted 2024-09-12T06:51:52.458 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:52.458 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=deleted 2024-09-12T06:51:52.458 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_image_id 2024-09-12T06:51:52.458 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_25 2024-09-12T06:51:52.458 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:52.458 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:52.458 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_25 2024-09-12T06:51:52.458 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_25 2024-09-12T06:51:52.458 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:52.493 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_25: (2) No such file or directory 2024-09-12T06:51:52.498 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_image_id= 2024-09-12T06:51:52.498 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:52.498 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test deleted = deleted 2024-09-12T06:51:52.498 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:52.498 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:52.498 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_26 2024-09-12T06:51:52.498 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_present cluster1 mirror image_26 deleted 2024-09-12T06:51:52.498 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:52.498 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:52.498 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_26 2024-09-12T06:51:52.498 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=deleted 2024-09-12T06:51:52.498 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:52.498 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:51:52.499 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:52.499 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_26 2024-09-12T06:51:52.499 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:52.499 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:52.499 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_26 2024-09-12T06:51:52.499 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_26 2024-09-12T06:51:52.499 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:52.534 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_26: (2) No such file or directory 2024-09-12T06:51:52.537 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image_id= 2024-09-12T06:51:52.537 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T06:51:52.537 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.1 2024-09-12T06:51:52.639 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_present cluster1 mirror image_26 deleted '' 2024-09-12T06:51:52.639 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:52.639 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:52.640 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_26 2024-09-12T06:51:52.640 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=deleted 2024-09-12T06:51:52.640 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:52.640 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=deleted 2024-09-12T06:51:52.640 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_image_id 2024-09-12T06:51:52.640 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_26 2024-09-12T06:51:52.640 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:52.640 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:52.640 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_26 2024-09-12T06:51:52.640 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_26 2024-09-12T06:51:52.640 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:52.674 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_26: (2) No such file or directory 2024-09-12T06:51:52.678 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_image_id= 2024-09-12T06:51:52.678 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:52.679 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test deleted = deleted 2024-09-12T06:51:52.679 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:52.679 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:52.679 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_27 2024-09-12T06:51:52.679 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_present cluster1 mirror image_27 deleted 2024-09-12T06:51:52.679 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:52.679 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:52.679 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_27 2024-09-12T06:51:52.679 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=deleted 2024-09-12T06:51:52.679 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:52.679 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:51:52.679 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:52.679 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_27 2024-09-12T06:51:52.680 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:52.680 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:52.680 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_27 2024-09-12T06:51:52.680 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_27 2024-09-12T06:51:52.680 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:52.714 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_27: (2) No such file or directory 2024-09-12T06:51:52.717 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image_id= 2024-09-12T06:51:52.717 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T06:51:52.717 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.1 2024-09-12T06:51:52.819 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_present cluster1 mirror image_27 deleted '' 2024-09-12T06:51:52.819 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:52.819 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:52.819 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_27 2024-09-12T06:51:52.819 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=deleted 2024-09-12T06:51:52.819 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:52.819 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=deleted 2024-09-12T06:51:52.819 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_image_id 2024-09-12T06:51:52.819 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_27 2024-09-12T06:51:52.819 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:52.820 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:52.820 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_27 2024-09-12T06:51:52.820 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_27 2024-09-12T06:51:52.820 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:52.854 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_27: (2) No such file or directory 2024-09-12T06:51:52.859 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_image_id= 2024-09-12T06:51:52.859 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:52.859 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test deleted = deleted 2024-09-12T06:51:52.859 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:52.859 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:52.859 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_28 2024-09-12T06:51:52.859 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_present cluster1 mirror image_28 deleted 2024-09-12T06:51:52.859 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:52.859 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:52.860 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_28 2024-09-12T06:51:52.860 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=deleted 2024-09-12T06:51:52.860 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:52.860 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:51:52.860 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:52.860 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_28 2024-09-12T06:51:52.860 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:52.860 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:52.860 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_28 2024-09-12T06:51:52.860 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_28 2024-09-12T06:51:52.860 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:52.894 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_28: (2) No such file or directory 2024-09-12T06:51:52.897 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image_id= 2024-09-12T06:51:52.898 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T06:51:52.898 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.1 2024-09-12T06:51:53.000 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_present cluster1 mirror image_28 deleted '' 2024-09-12T06:51:53.000 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:53.000 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:53.000 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_28 2024-09-12T06:51:53.000 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=deleted 2024-09-12T06:51:53.000 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:53.000 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=deleted 2024-09-12T06:51:53.000 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_image_id 2024-09-12T06:51:53.000 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_28 2024-09-12T06:51:53.000 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:53.000 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:53.001 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_28 2024-09-12T06:51:53.001 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_28 2024-09-12T06:51:53.001 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:53.039 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_28: (2) No such file or directory 2024-09-12T06:51:53.043 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_image_id= 2024-09-12T06:51:53.044 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:53.044 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test deleted = deleted 2024-09-12T06:51:53.044 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:53.044 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:53.044 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_29 2024-09-12T06:51:53.044 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_present cluster1 mirror image_29 deleted 2024-09-12T06:51:53.044 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:53.044 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:53.044 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_29 2024-09-12T06:51:53.044 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=deleted 2024-09-12T06:51:53.044 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:53.044 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:51:53.045 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:53.045 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_29 2024-09-12T06:51:53.045 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:53.045 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:53.045 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_29 2024-09-12T06:51:53.045 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_29 2024-09-12T06:51:53.045 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:53.079 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_29: (2) No such file or directory 2024-09-12T06:51:53.082 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image_id= 2024-09-12T06:51:53.082 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T06:51:53.082 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.1 2024-09-12T06:51:53.184 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_present cluster1 mirror image_29 deleted '' 2024-09-12T06:51:53.184 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:53.184 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:53.184 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_29 2024-09-12T06:51:53.184 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=deleted 2024-09-12T06:51:53.185 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:53.185 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=deleted 2024-09-12T06:51:53.185 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_image_id 2024-09-12T06:51:53.185 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_29 2024-09-12T06:51:53.185 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:53.185 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:53.185 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_29 2024-09-12T06:51:53.185 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_29 2024-09-12T06:51:53.185 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:53.219 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_29: (2) No such file or directory 2024-09-12T06:51:53.223 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_image_id= 2024-09-12T06:51:53.223 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:53.223 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test deleted = deleted 2024-09-12T06:51:53.223 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:53.224 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:53.224 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_30 2024-09-12T06:51:53.224 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_present cluster1 mirror image_30 deleted 2024-09-12T06:51:53.224 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:53.224 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:53.224 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_30 2024-09-12T06:51:53.224 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=deleted 2024-09-12T06:51:53.224 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:53.224 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:51:53.224 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:53.224 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_30 2024-09-12T06:51:53.224 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:53.224 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:53.224 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_30 2024-09-12T06:51:53.225 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_30 2024-09-12T06:51:53.225 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:53.260 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_30: (2) No such file or directory 2024-09-12T06:51:53.263 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image_id= 2024-09-12T06:51:53.263 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T06:51:53.263 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.1 2024-09-12T06:51:53.365 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_present cluster1 mirror image_30 deleted '' 2024-09-12T06:51:53.365 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:53.365 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:53.365 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_30 2024-09-12T06:51:53.365 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=deleted 2024-09-12T06:51:53.365 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:53.365 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=deleted 2024-09-12T06:51:53.366 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_image_id 2024-09-12T06:51:53.366 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_30 2024-09-12T06:51:53.366 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:53.366 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:53.366 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_30 2024-09-12T06:51:53.366 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_30 2024-09-12T06:51:53.366 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:53.401 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_30: (2) No such file or directory 2024-09-12T06:51:53.406 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_image_id= 2024-09-12T06:51:53.406 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:53.406 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test deleted = deleted 2024-09-12T06:51:53.406 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:53.406 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:53.406 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_31 2024-09-12T06:51:53.406 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_present cluster1 mirror image_31 deleted 2024-09-12T06:51:53.406 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:53.406 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:53.406 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_31 2024-09-12T06:51:53.406 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=deleted 2024-09-12T06:51:53.407 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:53.407 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:51:53.407 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:53.407 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_31 2024-09-12T06:51:53.407 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:53.407 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:53.407 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_31 2024-09-12T06:51:53.407 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_31 2024-09-12T06:51:53.407 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:53.442 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_31: (2) No such file or directory 2024-09-12T06:51:53.446 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image_id= 2024-09-12T06:51:53.446 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T06:51:53.446 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.1 2024-09-12T06:51:53.547 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_present cluster1 mirror image_31 deleted '' 2024-09-12T06:51:53.548 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:53.548 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:53.548 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_31 2024-09-12T06:51:53.548 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=deleted 2024-09-12T06:51:53.548 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:53.548 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=deleted 2024-09-12T06:51:53.548 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_image_id 2024-09-12T06:51:53.548 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_31 2024-09-12T06:51:53.548 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:53.548 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:53.548 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_31 2024-09-12T06:51:53.549 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_31 2024-09-12T06:51:53.549 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:53.584 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_31: (2) No such file or directory 2024-09-12T06:51:53.589 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_image_id= 2024-09-12T06:51:53.589 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:53.589 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test deleted = deleted 2024-09-12T06:51:53.589 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:53.589 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:53.589 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_32 2024-09-12T06:51:53.589 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_present cluster1 mirror image_32 deleted 2024-09-12T06:51:53.589 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:53.589 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:53.589 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_32 2024-09-12T06:51:53.590 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=deleted 2024-09-12T06:51:53.590 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:53.590 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:51:53.590 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:53.590 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_32 2024-09-12T06:51:53.590 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:53.590 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:53.590 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_32 2024-09-12T06:51:53.590 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_32 2024-09-12T06:51:53.590 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:53.624 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_32: (2) No such file or directory 2024-09-12T06:51:53.627 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image_id= 2024-09-12T06:51:53.627 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T06:51:53.627 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.1 2024-09-12T06:51:53.729 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_present cluster1 mirror image_32 deleted '' 2024-09-12T06:51:53.729 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:53.729 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:53.729 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_32 2024-09-12T06:51:53.729 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=deleted 2024-09-12T06:51:53.729 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:53.729 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=deleted 2024-09-12T06:51:53.730 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_image_id 2024-09-12T06:51:53.730 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_32 2024-09-12T06:51:53.730 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:53.730 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:53.730 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_32 2024-09-12T06:51:53.730 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_32 2024-09-12T06:51:53.730 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:53.764 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_32: (2) No such file or directory 2024-09-12T06:51:53.768 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_image_id= 2024-09-12T06:51:53.769 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:53.769 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test deleted = deleted 2024-09-12T06:51:53.769 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:53.769 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:53.769 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_33 2024-09-12T06:51:53.769 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_present cluster1 mirror image_33 deleted 2024-09-12T06:51:53.769 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:53.769 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:53.769 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_33 2024-09-12T06:51:53.769 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=deleted 2024-09-12T06:51:53.769 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:53.769 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:51:53.769 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:53.770 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_33 2024-09-12T06:51:53.770 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:53.770 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:53.770 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_33 2024-09-12T06:51:53.770 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_33 2024-09-12T06:51:53.770 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:53.803 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_33: (2) No such file or directory 2024-09-12T06:51:53.806 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image_id= 2024-09-12T06:51:53.807 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T06:51:53.807 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.1 2024-09-12T06:51:53.909 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_present cluster1 mirror image_33 deleted '' 2024-09-12T06:51:53.909 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:53.909 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:53.909 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_33 2024-09-12T06:51:53.909 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=deleted 2024-09-12T06:51:53.909 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:53.909 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=deleted 2024-09-12T06:51:53.909 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_image_id 2024-09-12T06:51:53.909 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_33 2024-09-12T06:51:53.909 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:53.909 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:53.910 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_33 2024-09-12T06:51:53.910 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_33 2024-09-12T06:51:53.910 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:53.944 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_33: (2) No such file or directory 2024-09-12T06:51:53.949 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_image_id= 2024-09-12T06:51:53.949 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:53.949 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test deleted = deleted 2024-09-12T06:51:53.949 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:53.949 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:53.949 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_34 2024-09-12T06:51:53.949 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_present cluster1 mirror image_34 deleted 2024-09-12T06:51:53.949 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:53.949 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:53.949 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_34 2024-09-12T06:51:53.949 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=deleted 2024-09-12T06:51:53.949 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:53.950 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:51:53.950 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:53.950 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_34 2024-09-12T06:51:53.950 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:53.950 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:53.950 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_34 2024-09-12T06:51:53.950 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_34 2024-09-12T06:51:53.950 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:53.984 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_34: (2) No such file or directory 2024-09-12T06:51:53.986 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image_id= 2024-09-12T06:51:53.987 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T06:51:53.987 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.1 2024-09-12T06:51:54.089 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_present cluster1 mirror image_34 deleted '' 2024-09-12T06:51:54.089 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:54.089 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:54.089 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_34 2024-09-12T06:51:54.089 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=deleted 2024-09-12T06:51:54.089 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:54.089 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=deleted 2024-09-12T06:51:54.089 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_image_id 2024-09-12T06:51:54.089 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_34 2024-09-12T06:51:54.089 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:54.089 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:54.089 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_34 2024-09-12T06:51:54.090 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_34 2024-09-12T06:51:54.090 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:54.124 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_34: (2) No such file or directory 2024-09-12T06:51:54.128 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_image_id= 2024-09-12T06:51:54.129 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:54.129 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test deleted = deleted 2024-09-12T06:51:54.129 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:54.129 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:54.129 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_35 2024-09-12T06:51:54.129 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_present cluster1 mirror image_35 deleted 2024-09-12T06:51:54.129 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:54.129 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:54.129 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_35 2024-09-12T06:51:54.129 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=deleted 2024-09-12T06:51:54.129 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:54.129 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:51:54.129 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:54.130 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_35 2024-09-12T06:51:54.130 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:54.130 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:54.130 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_35 2024-09-12T06:51:54.130 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_35 2024-09-12T06:51:54.130 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:54.164 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_35: (2) No such file or directory 2024-09-12T06:51:54.168 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image_id= 2024-09-12T06:51:54.168 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T06:51:54.168 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.1 2024-09-12T06:51:54.270 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_present cluster1 mirror image_35 deleted '' 2024-09-12T06:51:54.270 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:54.270 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:54.270 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_35 2024-09-12T06:51:54.270 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=deleted 2024-09-12T06:51:54.270 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:54.270 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=deleted 2024-09-12T06:51:54.270 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_image_id 2024-09-12T06:51:54.270 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_35 2024-09-12T06:51:54.270 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:54.271 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:54.271 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_35 2024-09-12T06:51:54.271 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_35 2024-09-12T06:51:54.271 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:54.306 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_35: (2) No such file or directory 2024-09-12T06:51:54.310 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_image_id= 2024-09-12T06:51:54.310 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:54.310 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test deleted = deleted 2024-09-12T06:51:54.311 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:54.311 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:54.311 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_36 2024-09-12T06:51:54.311 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_present cluster1 mirror image_36 deleted 2024-09-12T06:51:54.311 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:54.311 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:54.311 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_36 2024-09-12T06:51:54.311 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=deleted 2024-09-12T06:51:54.311 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:54.311 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:51:54.311 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:54.311 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_36 2024-09-12T06:51:54.311 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:54.312 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:54.312 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_36 2024-09-12T06:51:54.312 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_36 2024-09-12T06:51:54.312 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:54.346 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_36: (2) No such file or directory 2024-09-12T06:51:54.349 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image_id= 2024-09-12T06:51:54.349 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T06:51:54.349 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.1 2024-09-12T06:51:54.451 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_present cluster1 mirror image_36 deleted '' 2024-09-12T06:51:54.451 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:54.451 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:54.451 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_36 2024-09-12T06:51:54.451 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=deleted 2024-09-12T06:51:54.451 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:54.451 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=deleted 2024-09-12T06:51:54.451 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_image_id 2024-09-12T06:51:54.452 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_36 2024-09-12T06:51:54.452 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:54.452 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:54.452 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_36 2024-09-12T06:51:54.452 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_36 2024-09-12T06:51:54.452 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:54.487 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_36: (2) No such file or directory 2024-09-12T06:51:54.492 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_image_id= 2024-09-12T06:51:54.492 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:54.492 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test deleted = deleted 2024-09-12T06:51:54.492 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:54.492 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:54.492 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_37 2024-09-12T06:51:54.492 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_present cluster1 mirror image_37 deleted 2024-09-12T06:51:54.493 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:54.493 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:54.493 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_37 2024-09-12T06:51:54.493 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=deleted 2024-09-12T06:51:54.493 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:54.493 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:51:54.493 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:54.493 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_37 2024-09-12T06:51:54.493 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:54.493 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:54.493 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_37 2024-09-12T06:51:54.493 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_37 2024-09-12T06:51:54.493 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:54.527 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_37: (2) No such file or directory 2024-09-12T06:51:54.531 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image_id= 2024-09-12T06:51:54.531 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T06:51:54.531 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.1 2024-09-12T06:51:54.633 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_present cluster1 mirror image_37 deleted '' 2024-09-12T06:51:54.633 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:54.633 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:54.633 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_37 2024-09-12T06:51:54.633 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=deleted 2024-09-12T06:51:54.633 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:54.633 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=deleted 2024-09-12T06:51:54.634 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_image_id 2024-09-12T06:51:54.634 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_37 2024-09-12T06:51:54.634 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:54.634 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:54.634 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_37 2024-09-12T06:51:54.634 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_37 2024-09-12T06:51:54.634 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:54.668 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_37: (2) No such file or directory 2024-09-12T06:51:54.673 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_image_id= 2024-09-12T06:51:54.673 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:54.673 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test deleted = deleted 2024-09-12T06:51:54.673 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:54.673 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:54.673 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_38 2024-09-12T06:51:54.673 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_present cluster1 mirror image_38 deleted 2024-09-12T06:51:54.673 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:54.674 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:54.674 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_38 2024-09-12T06:51:54.674 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=deleted 2024-09-12T06:51:54.674 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:54.674 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:51:54.674 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:54.674 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_38 2024-09-12T06:51:54.674 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:54.674 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:54.674 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_38 2024-09-12T06:51:54.674 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_38 2024-09-12T06:51:54.674 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:54.707 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_38: (2) No such file or directory 2024-09-12T06:51:54.710 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image_id= 2024-09-12T06:51:54.710 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T06:51:54.711 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.1 2024-09-12T06:51:54.812 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_present cluster1 mirror image_38 deleted '' 2024-09-12T06:51:54.813 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:54.813 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:54.813 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_38 2024-09-12T06:51:54.813 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=deleted 2024-09-12T06:51:54.813 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:54.813 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=deleted 2024-09-12T06:51:54.813 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_image_id 2024-09-12T06:51:54.813 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_38 2024-09-12T06:51:54.813 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:54.813 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:54.813 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_38 2024-09-12T06:51:54.814 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_38 2024-09-12T06:51:54.966 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:54.969 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_38: (2) No such file or directory 2024-09-12T06:51:54.969 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_image_id= 2024-09-12T06:51:54.969 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:54.969 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test deleted = deleted 2024-09-12T06:51:54.969 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:54.969 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:54.969 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_39 2024-09-12T06:51:54.970 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_present cluster1 mirror image_39 deleted 2024-09-12T06:51:54.970 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:54.970 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:54.970 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_39 2024-09-12T06:51:54.970 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=deleted 2024-09-12T06:51:54.970 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:54.970 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:51:54.970 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:54.970 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_39 2024-09-12T06:51:54.970 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:54.970 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:54.970 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_39 2024-09-12T06:51:54.970 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_39 2024-09-12T06:51:54.970 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:54.970 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_39: (2) No such file or directory 2024-09-12T06:51:54.971 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image_id= 2024-09-12T06:51:54.971 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T06:51:54.971 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.1 2024-09-12T06:51:54.993 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_present cluster1 mirror image_39 deleted '' 2024-09-12T06:51:54.993 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:54.993 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:54.993 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_39 2024-09-12T06:51:54.994 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=deleted 2024-09-12T06:51:54.994 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:54.994 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=deleted 2024-09-12T06:51:54.994 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_image_id 2024-09-12T06:51:54.994 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_39 2024-09-12T06:51:54.994 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:54.994 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:54.994 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_39 2024-09-12T06:51:54.994 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_39 2024-09-12T06:51:54.994 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:55.031 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_39: (2) No such file or directory 2024-09-12T06:51:55.036 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_image_id= 2024-09-12T06:51:55.036 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:55.036 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test deleted = deleted 2024-09-12T06:51:55.036 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:55.036 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:55.037 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_40 2024-09-12T06:51:55.037 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_present cluster1 mirror image_40 deleted 2024-09-12T06:51:55.037 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:55.037 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:55.037 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_40 2024-09-12T06:51:55.037 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=deleted 2024-09-12T06:51:55.037 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:55.037 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:51:55.037 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:55.037 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_40 2024-09-12T06:51:55.037 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:55.037 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:55.037 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_40 2024-09-12T06:51:55.037 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_40 2024-09-12T06:51:55.037 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:55.075 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_40: (2) No such file or directory 2024-09-12T06:51:55.078 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image_id= 2024-09-12T06:51:55.078 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T06:51:55.078 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.1 2024-09-12T06:51:55.180 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_present cluster1 mirror image_40 deleted '' 2024-09-12T06:51:55.180 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:55.180 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:55.180 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_40 2024-09-12T06:51:55.180 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=deleted 2024-09-12T06:51:55.180 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:55.180 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=deleted 2024-09-12T06:51:55.181 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_image_id 2024-09-12T06:51:55.181 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_40 2024-09-12T06:51:55.181 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:55.181 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:55.181 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_40 2024-09-12T06:51:55.181 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_40 2024-09-12T06:51:55.181 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:55.216 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_40: (2) No such file or directory 2024-09-12T06:51:55.221 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_image_id= 2024-09-12T06:51:55.221 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:55.221 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test deleted = deleted 2024-09-12T06:51:55.221 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:55.221 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:55.221 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_41 2024-09-12T06:51:55.221 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_present cluster1 mirror image_41 deleted 2024-09-12T06:51:55.221 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:55.221 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:55.222 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_41 2024-09-12T06:51:55.222 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=deleted 2024-09-12T06:51:55.222 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:55.222 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:51:55.222 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:55.222 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_41 2024-09-12T06:51:55.222 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:55.222 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:55.222 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_41 2024-09-12T06:51:55.222 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_41 2024-09-12T06:51:55.222 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:55.256 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_41: (2) No such file or directory 2024-09-12T06:51:55.259 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image_id= 2024-09-12T06:51:55.259 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T06:51:55.259 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.1 2024-09-12T06:51:55.361 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_present cluster1 mirror image_41 deleted '' 2024-09-12T06:51:55.361 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:55.361 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:55.361 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_41 2024-09-12T06:51:55.362 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=deleted 2024-09-12T06:51:55.362 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:55.362 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=deleted 2024-09-12T06:51:55.362 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_image_id 2024-09-12T06:51:55.362 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_41 2024-09-12T06:51:55.362 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:55.362 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:55.362 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_41 2024-09-12T06:51:55.362 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_41 2024-09-12T06:51:55.362 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:55.397 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_41: (2) No such file or directory 2024-09-12T06:51:55.402 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_image_id= 2024-09-12T06:51:55.402 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:55.402 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test deleted = deleted 2024-09-12T06:51:55.402 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:55.402 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:55.403 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_42 2024-09-12T06:51:55.403 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_present cluster1 mirror image_42 deleted 2024-09-12T06:51:55.403 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:55.403 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:55.403 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_42 2024-09-12T06:51:55.403 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=deleted 2024-09-12T06:51:55.403 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:55.403 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:51:55.403 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:55.403 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_42 2024-09-12T06:51:55.403 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:55.403 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:55.403 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_42 2024-09-12T06:51:55.403 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_42 2024-09-12T06:51:55.404 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:55.437 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_42: (2) No such file or directory 2024-09-12T06:51:55.441 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image_id= 2024-09-12T06:51:55.441 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T06:51:55.441 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.1 2024-09-12T06:51:55.543 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_present cluster1 mirror image_42 deleted '' 2024-09-12T06:51:55.543 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:55.543 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:55.543 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_42 2024-09-12T06:51:55.543 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=deleted 2024-09-12T06:51:55.543 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:55.543 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=deleted 2024-09-12T06:51:55.543 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_image_id 2024-09-12T06:51:55.543 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_42 2024-09-12T06:51:55.543 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:55.543 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:55.544 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_42 2024-09-12T06:51:55.544 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_42 2024-09-12T06:51:55.544 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:55.578 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_42: (2) No such file or directory 2024-09-12T06:51:55.582 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_image_id= 2024-09-12T06:51:55.582 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:55.582 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test deleted = deleted 2024-09-12T06:51:55.583 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:55.583 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:55.583 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_43 2024-09-12T06:51:55.583 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_present cluster1 mirror image_43 deleted 2024-09-12T06:51:55.583 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:55.583 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:55.583 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_43 2024-09-12T06:51:55.583 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=deleted 2024-09-12T06:51:55.583 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:55.583 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:51:55.583 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:55.583 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_43 2024-09-12T06:51:55.583 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:55.583 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:55.584 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_43 2024-09-12T06:51:55.584 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_43 2024-09-12T06:51:55.584 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:55.622 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_43: (2) No such file or directory 2024-09-12T06:51:55.622 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image_id= 2024-09-12T06:51:55.622 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T06:51:55.622 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.1 2024-09-12T06:51:55.722 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_present cluster1 mirror image_43 deleted '' 2024-09-12T06:51:55.722 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:55.723 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:55.723 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_43 2024-09-12T06:51:55.723 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=deleted 2024-09-12T06:51:55.723 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:55.723 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=deleted 2024-09-12T06:51:55.723 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_image_id 2024-09-12T06:51:55.723 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_43 2024-09-12T06:51:55.723 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:55.723 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:55.723 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_43 2024-09-12T06:51:55.723 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_43 2024-09-12T06:51:55.723 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:55.758 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_43: (2) No such file or directory 2024-09-12T06:51:55.763 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_image_id= 2024-09-12T06:51:55.763 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:55.763 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test deleted = deleted 2024-09-12T06:51:55.763 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:55.763 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:55.763 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_44 2024-09-12T06:51:55.763 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_present cluster1 mirror image_44 deleted 2024-09-12T06:51:55.763 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:55.764 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:55.764 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_44 2024-09-12T06:51:55.764 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=deleted 2024-09-12T06:51:55.764 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:55.764 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:51:55.764 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:55.764 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_44 2024-09-12T06:51:55.764 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:55.764 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:55.764 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_44 2024-09-12T06:51:55.764 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_44 2024-09-12T06:51:55.764 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:55.799 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_44: (2) No such file or directory 2024-09-12T06:51:55.802 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image_id= 2024-09-12T06:51:55.802 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T06:51:55.802 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.1 2024-09-12T06:51:55.904 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_present cluster1 mirror image_44 deleted '' 2024-09-12T06:51:55.904 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:55.904 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:55.904 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_44 2024-09-12T06:51:55.904 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=deleted 2024-09-12T06:51:55.905 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:55.905 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=deleted 2024-09-12T06:51:55.905 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_image_id 2024-09-12T06:51:55.905 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_44 2024-09-12T06:51:55.905 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:55.905 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:55.905 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_44 2024-09-12T06:51:55.905 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_44 2024-09-12T06:51:55.905 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:55.945 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_44: (2) No such file or directory 2024-09-12T06:51:55.950 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_image_id= 2024-09-12T06:51:55.950 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:55.950 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test deleted = deleted 2024-09-12T06:51:55.950 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:55.950 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:55.950 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_45 2024-09-12T06:51:55.950 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_present cluster1 mirror image_45 deleted 2024-09-12T06:51:55.950 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:55.950 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:55.950 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_45 2024-09-12T06:51:55.950 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=deleted 2024-09-12T06:51:55.950 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:55.950 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:51:55.951 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:55.951 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_45 2024-09-12T06:51:55.951 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:55.951 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:55.951 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_45 2024-09-12T06:51:55.951 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_45 2024-09-12T06:51:55.951 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:55.985 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_45: (2) No such file or directory 2024-09-12T06:51:55.988 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image_id= 2024-09-12T06:51:55.988 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T06:51:55.988 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.1 2024-09-12T06:51:56.090 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_present cluster1 mirror image_45 deleted '' 2024-09-12T06:51:56.090 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:56.090 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:56.090 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_45 2024-09-12T06:51:56.090 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=deleted 2024-09-12T06:51:56.091 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:56.091 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=deleted 2024-09-12T06:51:56.091 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_image_id 2024-09-12T06:51:56.091 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_45 2024-09-12T06:51:56.091 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:56.091 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:56.091 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_45 2024-09-12T06:51:56.091 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_45 2024-09-12T06:51:56.091 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:56.126 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_45: (2) No such file or directory 2024-09-12T06:51:56.131 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_image_id= 2024-09-12T06:51:56.131 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:56.131 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test deleted = deleted 2024-09-12T06:51:56.131 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:56.131 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:56.131 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_46 2024-09-12T06:51:56.131 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_present cluster1 mirror image_46 deleted 2024-09-12T06:51:56.131 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:56.131 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:56.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_46 2024-09-12T06:51:56.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=deleted 2024-09-12T06:51:56.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:56.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:51:56.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:56.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_46 2024-09-12T06:51:56.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:56.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:56.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_46 2024-09-12T06:51:56.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_46 2024-09-12T06:51:56.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:56.173 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_46: (2) No such file or directory 2024-09-12T06:51:56.176 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image_id= 2024-09-12T06:51:56.176 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T06:51:56.176 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.1 2024-09-12T06:51:56.278 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_present cluster1 mirror image_46 deleted '' 2024-09-12T06:51:56.278 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:56.278 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:56.279 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_46 2024-09-12T06:51:56.279 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=deleted 2024-09-12T06:51:56.279 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:56.279 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=deleted 2024-09-12T06:51:56.279 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_image_id 2024-09-12T06:51:56.279 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_46 2024-09-12T06:51:56.279 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:56.279 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:56.279 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_46 2024-09-12T06:51:56.279 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_46 2024-09-12T06:51:56.279 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:56.313 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_46: (2) No such file or directory 2024-09-12T06:51:56.318 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_image_id= 2024-09-12T06:51:56.318 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:56.318 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test deleted = deleted 2024-09-12T06:51:56.319 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:56.319 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:56.319 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_47 2024-09-12T06:51:56.319 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_present cluster1 mirror image_47 deleted 2024-09-12T06:51:56.319 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:56.319 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:56.319 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_47 2024-09-12T06:51:56.319 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=deleted 2024-09-12T06:51:56.319 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:56.319 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:51:56.319 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:56.320 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_47 2024-09-12T06:51:56.320 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:56.320 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:56.320 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_47 2024-09-12T06:51:56.320 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_47 2024-09-12T06:51:56.320 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:56.352 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_47: (2) No such file or directory 2024-09-12T06:51:56.356 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image_id= 2024-09-12T06:51:56.356 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T06:51:56.356 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.1 2024-09-12T06:51:56.458 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_present cluster1 mirror image_47 deleted '' 2024-09-12T06:51:56.458 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:56.459 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:56.459 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_47 2024-09-12T06:51:56.459 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=deleted 2024-09-12T06:51:56.459 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:56.459 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=deleted 2024-09-12T06:51:56.459 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_image_id 2024-09-12T06:51:56.459 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_47 2024-09-12T06:51:56.459 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:56.459 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:56.459 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_47 2024-09-12T06:51:56.460 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_47 2024-09-12T06:51:56.460 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:56.492 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_47: (2) No such file or directory 2024-09-12T06:51:56.497 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_image_id= 2024-09-12T06:51:56.497 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:56.497 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test deleted = deleted 2024-09-12T06:51:56.497 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:56.497 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:56.497 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_48 2024-09-12T06:51:56.497 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_present cluster1 mirror image_48 deleted 2024-09-12T06:51:56.497 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:56.497 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:56.498 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_48 2024-09-12T06:51:56.498 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=deleted 2024-09-12T06:51:56.498 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:56.498 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:51:56.498 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:56.498 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_48 2024-09-12T06:51:56.498 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:56.498 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:56.498 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_48 2024-09-12T06:51:56.498 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_48 2024-09-12T06:51:56.498 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:56.533 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_48: (2) No such file or directory 2024-09-12T06:51:56.536 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image_id= 2024-09-12T06:51:56.536 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T06:51:56.536 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.1 2024-09-12T06:51:56.639 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_present cluster1 mirror image_48 deleted '' 2024-09-12T06:51:56.639 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:56.639 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:56.640 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_48 2024-09-12T06:51:56.640 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=deleted 2024-09-12T06:51:56.640 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:56.640 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=deleted 2024-09-12T06:51:56.640 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_image_id 2024-09-12T06:51:56.640 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_48 2024-09-12T06:51:56.641 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:56.641 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:56.641 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_48 2024-09-12T06:51:56.641 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_48 2024-09-12T06:51:56.641 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:56.674 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_48: (2) No such file or directory 2024-09-12T06:51:56.679 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_image_id= 2024-09-12T06:51:56.679 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:56.679 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test deleted = deleted 2024-09-12T06:51:56.680 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:56.680 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:56.680 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_49 2024-09-12T06:51:56.680 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_present cluster1 mirror image_49 deleted 2024-09-12T06:51:56.680 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:56.680 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:56.680 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_49 2024-09-12T06:51:56.680 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=deleted 2024-09-12T06:51:56.680 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:56.681 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:51:56.681 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:56.681 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_49 2024-09-12T06:51:56.681 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:56.681 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:56.681 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_49 2024-09-12T06:51:56.681 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_49 2024-09-12T06:51:56.682 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:56.713 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_49: (2) No such file or directory 2024-09-12T06:51:56.716 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image_id= 2024-09-12T06:51:56.717 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T06:51:56.717 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.1 2024-09-12T06:51:56.818 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_present cluster1 mirror image_49 deleted '' 2024-09-12T06:51:56.818 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:56.818 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:56.818 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_49 2024-09-12T06:51:56.818 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=deleted 2024-09-12T06:51:56.818 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:56.819 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=deleted 2024-09-12T06:51:56.819 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_image_id 2024-09-12T06:51:56.819 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_49 2024-09-12T06:51:56.819 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:56.819 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:56.820 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_49 2024-09-12T06:51:56.820 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_49 2024-09-12T06:51:56.820 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:56.855 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_49: (2) No such file or directory 2024-09-12T06:51:56.862 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_image_id= 2024-09-12T06:51:56.862 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:56.862 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test deleted = deleted 2024-09-12T06:51:56.862 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:56.862 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T06:51:56.863 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_50 2024-09-12T06:51:56.863 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_present cluster1 mirror image_50 deleted 2024-09-12T06:51:56.863 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:56.863 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:56.863 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_50 2024-09-12T06:51:56.864 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=deleted 2024-09-12T06:51:56.864 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:56.864 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:51:56.864 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:56.864 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_50 2024-09-12T06:51:56.864 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:56.865 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:56.865 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_50 2024-09-12T06:51:56.865 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_50 2024-09-12T06:51:56.865 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:56.894 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_50: (2) No such file or directory 2024-09-12T06:51:56.897 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image_id= 2024-09-12T06:51:56.898 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T06:51:56.898 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.1 2024-09-12T06:51:57.001 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_present cluster1 mirror image_50 deleted '' 2024-09-12T06:51:57.001 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:57.001 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:57.001 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_50 2024-09-12T06:51:57.001 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=deleted 2024-09-12T06:51:57.001 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:51:57.002 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=deleted 2024-09-12T06:51:57.002 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_image_id 2024-09-12T06:51:57.002 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_50 2024-09-12T06:51:57.002 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:51:57.002 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:51:57.002 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_50 2024-09-12T06:51:57.002 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_50 2024-09-12T06:51:57.002 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:51:57.035 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_50: (2) No such file or directory 2024-09-12T06:51:57.038 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_image_id= 2024-09-12T06:51:57.038 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:57.038 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test deleted = deleted 2024-09-12T06:51:57.039 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:51:57.039 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ testlog 'TEST: delete images during bootstrap' 2024-09-12T06:51:57.039 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ tee -a /tmp/tmp.GzjHZGs9Z0/rbd-mirror.test.log 2024-09-12T06:51:57.039 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ date '+%F %T' 2024-09-12T06:51:57.040 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo 2024-09-12 06:51:57 TEST: delete images during bootstrap 2024-09-12T06:51:57.040 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:2024-09-12 06:51:57 TEST: delete images during bootstrap 2024-09-12T06:51:57.040 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_pool_mirror_mode cluster1 mirror image 2024-09-12T06:51:57.040 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:57.040 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:57.041 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local mode=image 2024-09-12T06:51:57.041 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster1 mirror pool enable mirror image 2024-09-12T06:51:57.076 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:note: changing mirroring mode from pool to image 2024-09-12T06:51:57.088 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_pool_mirror_mode cluster2 mirror image 2024-09-12T06:51:57.088 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:57.089 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:57.089 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local mode=image 2024-09-12T06:51:57.089 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 mirror pool enable mirror image 2024-09-12T06:51:57.124 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:note: changing mirroring mode from pool to image 2024-09-12T06:51:57.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ start_mirror cluster1 2024-09-12T06:51:57.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:51:57.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance 2024-09-12T06:51:57.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set_cluster_instance cluster1 cluster instance 2024-09-12T06:51:57.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local val=cluster1 2024-09-12T06:51:57.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster_var_name=cluster 2024-09-12T06:51:57.132 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local instance_var_name=instance 2024-09-12T06:51:57.133 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cluster=cluster1 2024-09-12T06:51:57.133 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=cluster1 2024-09-12T06:51:57.133 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' cluster1 = cluster1 ']' 2024-09-12T06:51:57.133 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ instance=0 2024-09-12T06:51:57.133 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval cluster=cluster1 2024-09-12T06:51:57.133 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:51:57.133 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ eval instance=0 2024-09-12T06:51:57.133 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:51:57.133 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:51:57.133 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd-mirror --cluster cluster1 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2024-09-12T06:51:57.270 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:2024-09-12T06:51:57.271+0000 7f0d4e5ae880 -1 open: failed to lock pidfile /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid because another process locked it': (11) Resource temporarily unavailable 2024-09-12T06:51:57.270 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=test 2024-09-12T06:51:57.270 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 1 10 2024-09-12T06:51:57.272 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 10` 2024-09-12T06:51:57.272 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_1 2024-09-12T06:51:57.272 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image cluster2 mirror image_1 512M 2024-09-12T06:51:57.272 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:57.272 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:51:57.272 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:57.272 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:51:57.272 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_1 2024-09-12T06:51:57.273 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:51:57.273 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local size=128 2024-09-12T06:51:57.273 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n 512M ']' 2024-09-12T06:51:57.273 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ size=512M 2024-09-12T06:51:57.273 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:51:57.273 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/image_1 2024-09-12T06:51:57.330 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ enable_mirror cluster2 mirror image_1 2024-09-12T06:51:57.330 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:57.330 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:57.330 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_1 2024-09-12T06:51:57.330 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local mode=journal 2024-09-12T06:51:57.330 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_1 journal 2024-09-12T06:51:57.378 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:Mirroring enabled 2024-09-12T06:51:57.383 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 info mirror/image_1 2024-09-12T06:51:57.424 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:rbd image 'image_1': 2024-09-12T06:51:57.424 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: size 512 MiB in 128 objects 2024-09-12T06:51:57.424 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: order 22 (4 MiB objects) 2024-09-12T06:51:57.424 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: snapshot_count: 0 2024-09-12T06:51:57.424 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: id: 1995a6f2be5c 2024-09-12T06:51:57.424 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: block_name_prefix: rbd_data.1995a6f2be5c 2024-09-12T06:51:57.424 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: format: 2 2024-09-12T06:51:57.425 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: features: layering, exclusive-lock, journaling 2024-09-12T06:51:57.425 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: op_features: 2024-09-12T06:51:57.425 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: flags: 2024-09-12T06:51:57.425 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: create_timestamp: Thu Sep 12 06:51:57 2024 2024-09-12T06:51:57.425 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: access_timestamp: Thu Sep 12 06:51:57 2024 2024-09-12T06:51:57.425 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: modify_timestamp: Thu Sep 12 06:51:57 2024 2024-09-12T06:51:57.425 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: journal: 1995a6f2be5c 2024-09-12T06:51:57.425 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring state: enabled 2024-09-12T06:51:57.425 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring mode: journal 2024-09-12T06:51:57.425 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring global id: 868fe9d6-4b30-4720-a935-df201615dda4 2024-09-12T06:51:57.425 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring primary: true 2024-09-12T06:51:57.427 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ stress_write_image cluster2 mirror image_1 2024-09-12T06:51:57.428 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:51:57.428 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:51:57.428 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_1 2024-09-12T06:51:57.428 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T06:51:57.429 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local duration=13 2024-09-12T06:51:57.429 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set +e 2024-09-12T06:51:57.429 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_1 --debug-rbd=20 --debug-journaler=20 2024-09-12T06:51:59.146 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:100 IOs, 29330432 bytes 2024-09-12T06:52:05.150 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:200 IOs, 149060608 bytes 2024-09-12T06:52:09.636 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:300 IOs, 219100160 bytes 2024-09-12T06:52:10.438 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ error_code=124 2024-09-12T06:52:10.438 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set -e 2024-09-12T06:52:10.438 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T06:52:10.438 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:52:10.438 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_present cluster1 mirror image_1 present 2024-09-12T06:52:10.438 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:52:10.438 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:52:10.438 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_1 2024-09-12T06:52:10.438 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=present 2024-09-12T06:52:10.438 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:52:10.438 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:52:10.439 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:52:10.439 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_1 2024-09-12T06:52:10.439 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:52:10.439 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:52:10.439 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_1 2024-09-12T06:52:10.439 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-12T06:52:10.439 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:52:10.485 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image_id=10a86745bfc1 2024-09-12T06:52:10.485 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T06:52:10.485 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.1 2024-09-12T06:52:10.587 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_present cluster1 mirror image_1 present 10a86745bfc1 2024-09-12T06:52:10.588 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:52:10.588 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:52:10.588 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_1 2024-09-12T06:52:10.588 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=present 2024-09-12T06:52:10.588 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id=10a86745bfc1 2024-09-12T06:52:10.588 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=deleted 2024-09-12T06:52:10.588 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_image_id 2024-09-12T06:52:10.588 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_1 2024-09-12T06:52:10.588 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:52:10.588 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:52:10.588 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_1 2024-09-12T06:52:10.588 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-12T06:52:10.588 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:52:10.636 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_image_id=10a86745bfc1 2024-09-12T06:52:10.636 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 10a86745bfc1 2024-09-12T06:52:10.636 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -z 10a86745bfc1 -o 10a86745bfc1 = 10a86745bfc1 2024-09-12T06:52:10.636 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_state=present 2024-09-12T06:52:10.636 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test present = present 2024-09-12T06:52:10.636 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:52:10.636 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ disable_mirror cluster2 mirror image_1 2024-09-12T06:52:10.637 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:52:10.637 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:52:10.637 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_1 2024-09-12T06:52:10.637 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_1 2024-09-12T06:52:38.389 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:Mirroring disabled 2024-09-12T06:52:38.396 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_present cluster1 mirror image_1 deleted 2024-09-12T06:52:38.396 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:52:38.396 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:52:38.396 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_1 2024-09-12T06:52:38.396 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=deleted 2024-09-12T06:52:38.396 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:52:38.397 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:52:38.397 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:52:38.397 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_1 2024-09-12T06:52:38.397 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:52:38.397 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:52:38.397 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_1 2024-09-12T06:52:38.397 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-12T06:52:38.397 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:52:38.432 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-09-12T06:52:38.435 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image_id= 2024-09-12T06:52:38.435 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T06:52:38.435 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.1 2024-09-12T06:52:38.537 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_present cluster1 mirror image_1 deleted '' 2024-09-12T06:52:38.537 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:52:38.537 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:52:38.537 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_1 2024-09-12T06:52:38.537 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=deleted 2024-09-12T06:52:38.538 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:52:38.538 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=deleted 2024-09-12T06:52:38.538 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_image_id 2024-09-12T06:52:38.538 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_1 2024-09-12T06:52:38.538 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:52:38.538 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:52:38.538 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_1 2024-09-12T06:52:38.538 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-12T06:52:38.538 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:52:38.573 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-09-12T06:52:38.576 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_image_id= 2024-09-12T06:52:38.576 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:52:38.577 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test deleted = deleted 2024-09-12T06:52:38.577 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:52:38.577 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ purge_snapshots cluster2 mirror image_1 2024-09-12T06:52:38.577 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:52:38.577 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:52:38.577 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_1 2024-09-12T06:52:38.577 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 snap purge mirror/image_1 2024-09-12T06:52:38.623 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image_retry cluster2 mirror image_1 2024-09-12T06:52:38.623 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:52:38.623 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:52:38.623 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_1 2024-09-12T06:52:38.623 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T06:52:38.623 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:52:38.623 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image cluster2 mirror image_1 2024-09-12T06:52:38.624 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:52:38.624 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:52:38.624 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_1 2024-09-12T06:52:38.624 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2024-09-12T06:52:38.670 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2024-09-12T06:52:38.890 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-12T06:52:38.894 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:52:38.894 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 10` 2024-09-12T06:52:38.894 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_2 2024-09-12T06:52:38.894 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image cluster2 mirror image_2 512M 2024-09-12T06:52:38.894 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:52:38.894 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:52:38.894 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:52:38.894 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:52:38.894 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_2 2024-09-12T06:52:38.894 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:52:38.894 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local size=128 2024-09-12T06:52:38.895 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n 512M ']' 2024-09-12T06:52:38.895 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ size=512M 2024-09-12T06:52:38.895 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:52:38.895 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/image_2 2024-09-12T06:52:38.953 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ enable_mirror cluster2 mirror image_2 2024-09-12T06:52:38.953 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:52:38.953 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:52:38.953 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_2 2024-09-12T06:52:38.953 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local mode=journal 2024-09-12T06:52:38.953 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_2 journal 2024-09-12T06:52:39.001 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:Mirroring enabled 2024-09-12T06:52:39.007 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 info mirror/image_2 2024-09-12T06:52:39.047 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:rbd image 'image_2': 2024-09-12T06:52:39.047 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: size 512 MiB in 128 objects 2024-09-12T06:52:39.047 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: order 22 (4 MiB objects) 2024-09-12T06:52:39.047 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: snapshot_count: 0 2024-09-12T06:52:39.047 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: id: 19ad50b40a28 2024-09-12T06:52:39.047 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: block_name_prefix: rbd_data.19ad50b40a28 2024-09-12T06:52:39.047 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: format: 2 2024-09-12T06:52:39.047 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: features: layering, exclusive-lock, journaling 2024-09-12T06:52:39.048 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: op_features: 2024-09-12T06:52:39.048 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: flags: 2024-09-12T06:52:39.048 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: create_timestamp: Thu Sep 12 06:52:38 2024 2024-09-12T06:52:39.048 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: access_timestamp: Thu Sep 12 06:52:38 2024 2024-09-12T06:52:39.048 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: modify_timestamp: Thu Sep 12 06:52:38 2024 2024-09-12T06:52:39.048 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: journal: 19ad50b40a28 2024-09-12T06:52:39.048 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring state: enabled 2024-09-12T06:52:39.048 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring mode: journal 2024-09-12T06:52:39.048 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring global id: 34d597cb-c49c-452f-9203-a28ecf9e9ed8 2024-09-12T06:52:39.048 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring primary: true 2024-09-12T06:52:39.050 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ stress_write_image cluster2 mirror image_2 2024-09-12T06:52:39.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:52:39.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:52:39.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_2 2024-09-12T06:52:39.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T06:52:39.053 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local duration=13 2024-09-12T06:52:39.053 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set +e 2024-09-12T06:52:39.053 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_2 --debug-rbd=20 --debug-journaler=20 2024-09-12T06:52:40.460 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:100 IOs, 23056384 bytes 2024-09-12T06:52:40.667 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:200 IOs, 23670784 bytes 2024-09-12T06:52:43.272 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:300 IOs, 71363584 bytes 2024-09-12T06:52:46.572 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:400 IOs, 125058048 bytes 2024-09-12T06:52:51.219 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:500 IOs, 201938944 bytes 2024-09-12T06:52:52.063 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ error_code=124 2024-09-12T06:52:52.063 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set -e 2024-09-12T06:52:52.063 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T06:52:52.063 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:52:52.063 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_present cluster1 mirror image_2 present 2024-09-12T06:52:52.063 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:52:52.063 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:52:52.064 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_2 2024-09-12T06:52:52.064 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=present 2024-09-12T06:52:52.064 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:52:52.064 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:52:52.064 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:52:52.064 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_2 2024-09-12T06:52:52.064 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:52:52.064 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:52:52.064 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_2 2024-09-12T06:52:52.064 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-12T06:52:52.064 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:52:52.315 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image_id=10a8963b0abf 2024-09-12T06:52:52.315 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T06:52:52.315 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.1 2024-09-12T06:52:52.417 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_present cluster1 mirror image_2 present 10a8963b0abf 2024-09-12T06:52:52.417 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:52:52.417 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:52:52.417 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_2 2024-09-12T06:52:52.417 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=present 2024-09-12T06:52:52.417 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id=10a8963b0abf 2024-09-12T06:52:52.417 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=deleted 2024-09-12T06:52:52.417 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_image_id 2024-09-12T06:52:52.417 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_2 2024-09-12T06:52:52.418 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:52:52.418 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:52:52.418 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_2 2024-09-12T06:52:52.418 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-12T06:52:52.418 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:52:52.465 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_image_id=10a8963b0abf 2024-09-12T06:52:52.465 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 10a8963b0abf 2024-09-12T06:52:52.466 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -z 10a8963b0abf -o 10a8963b0abf = 10a8963b0abf 2024-09-12T06:52:52.466 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_state=present 2024-09-12T06:52:52.466 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test present = present 2024-09-12T06:52:52.466 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:52:52.466 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ disable_mirror cluster2 mirror image_2 2024-09-12T06:52:52.466 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:52:52.466 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:52:52.466 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_2 2024-09-12T06:52:52.466 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_2 2024-09-12T06:53:22.198 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:Mirroring disabled 2024-09-12T06:53:22.205 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_present cluster1 mirror image_2 deleted 2024-09-12T06:53:22.206 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:53:22.206 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:53:22.206 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_2 2024-09-12T06:53:22.206 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=deleted 2024-09-12T06:53:22.206 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:53:22.206 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:53:22.206 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:53:22.206 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_2 2024-09-12T06:53:22.206 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:53:22.206 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:53:22.206 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_2 2024-09-12T06:53:22.206 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-12T06:53:22.207 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:53:22.242 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-09-12T06:53:22.245 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image_id= 2024-09-12T06:53:22.245 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T06:53:22.245 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.1 2024-09-12T06:53:22.347 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_present cluster1 mirror image_2 deleted '' 2024-09-12T06:53:22.347 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:53:22.347 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:53:22.347 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_2 2024-09-12T06:53:22.347 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=deleted 2024-09-12T06:53:22.347 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:53:22.347 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=deleted 2024-09-12T06:53:22.347 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_image_id 2024-09-12T06:53:22.348 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_2 2024-09-12T06:53:22.348 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:53:22.348 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:53:22.348 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_2 2024-09-12T06:53:22.348 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-12T06:53:22.348 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:53:22.383 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-09-12T06:53:22.387 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_image_id= 2024-09-12T06:53:22.387 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:53:22.387 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test deleted = deleted 2024-09-12T06:53:22.387 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:53:22.388 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ purge_snapshots cluster2 mirror image_2 2024-09-12T06:53:22.388 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:53:22.388 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:53:22.388 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_2 2024-09-12T06:53:22.388 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 snap purge mirror/image_2 2024-09-12T06:53:22.434 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image_retry cluster2 mirror image_2 2024-09-12T06:53:22.434 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:53:22.434 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:53:22.434 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_2 2024-09-12T06:53:22.434 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T06:53:22.434 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:53:22.435 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image cluster2 mirror image_2 2024-09-12T06:53:22.435 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:53:22.435 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:53:22.435 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_2 2024-09-12T06:53:22.435 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2024-09-12T06:53:22.483 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2024-09-12T06:53:22.704 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-12T06:53:22.708 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:53:22.709 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 10` 2024-09-12T06:53:22.709 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_3 2024-09-12T06:53:22.709 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image cluster2 mirror image_3 512M 2024-09-12T06:53:22.709 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:53:22.709 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:53:22.709 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:53:22.709 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:53:22.709 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_3 2024-09-12T06:53:22.709 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:53:22.709 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local size=128 2024-09-12T06:53:22.709 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n 512M ']' 2024-09-12T06:53:22.709 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ size=512M 2024-09-12T06:53:22.709 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:53:22.709 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/image_3 2024-09-12T06:53:22.766 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ enable_mirror cluster2 mirror image_3 2024-09-12T06:53:22.766 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:53:22.766 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:53:22.766 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_3 2024-09-12T06:53:22.766 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local mode=journal 2024-09-12T06:53:22.766 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_3 journal 2024-09-12T06:53:22.814 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:Mirroring enabled 2024-09-12T06:53:22.819 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 info mirror/image_3 2024-09-12T06:53:22.859 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:rbd image 'image_3': 2024-09-12T06:53:22.860 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: size 512 MiB in 128 objects 2024-09-12T06:53:22.860 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: order 22 (4 MiB objects) 2024-09-12T06:53:22.860 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: snapshot_count: 0 2024-09-12T06:53:22.860 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: id: 19c5d3d14a33 2024-09-12T06:53:22.860 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: block_name_prefix: rbd_data.19c5d3d14a33 2024-09-12T06:53:22.860 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: format: 2 2024-09-12T06:53:22.860 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: features: layering, exclusive-lock, journaling 2024-09-12T06:53:22.860 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: op_features: 2024-09-12T06:53:22.860 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: flags: 2024-09-12T06:53:22.860 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: create_timestamp: Thu Sep 12 06:53:22 2024 2024-09-12T06:53:22.860 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: access_timestamp: Thu Sep 12 06:53:22 2024 2024-09-12T06:53:22.860 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: modify_timestamp: Thu Sep 12 06:53:22 2024 2024-09-12T06:53:22.860 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: journal: 19c5d3d14a33 2024-09-12T06:53:22.860 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring state: enabled 2024-09-12T06:53:22.861 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring mode: journal 2024-09-12T06:53:22.861 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring global id: ebadaeb4-4dc2-4b7f-82a7-5d4686bd6da1 2024-09-12T06:53:22.861 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring primary: true 2024-09-12T06:53:22.863 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ stress_write_image cluster2 mirror image_3 2024-09-12T06:53:22.863 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:53:22.863 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:53:22.863 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_3 2024-09-12T06:53:22.863 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T06:53:22.865 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local duration=5 2024-09-12T06:53:22.865 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set +e 2024-09-12T06:53:22.865 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_3 --debug-rbd=20 --debug-journaler=20 2024-09-12T06:53:24.345 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:100 IOs, 20791296 bytes 2024-09-12T06:53:27.161 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:200 IOs, 62526464 bytes 2024-09-12T06:53:27.872 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ error_code=124 2024-09-12T06:53:27.872 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set -e 2024-09-12T06:53:27.872 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T06:53:27.872 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:53:27.873 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_present cluster1 mirror image_3 present 2024-09-12T06:53:27.873 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:53:27.873 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:53:27.873 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_3 2024-09-12T06:53:27.873 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=present 2024-09-12T06:53:27.873 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:53:27.873 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:53:27.873 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:53:27.873 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_3 2024-09-12T06:53:27.873 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:53:27.873 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:53:27.873 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_3 2024-09-12T06:53:27.874 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-12T06:53:27.874 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:53:27.920 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image_id=10a8f2f052cd 2024-09-12T06:53:27.921 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T06:53:27.921 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.1 2024-09-12T06:53:28.023 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_present cluster1 mirror image_3 present 10a8f2f052cd 2024-09-12T06:53:28.023 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:53:28.023 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:53:28.024 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_3 2024-09-12T06:53:28.024 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=present 2024-09-12T06:53:28.024 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id=10a8f2f052cd 2024-09-12T06:53:28.024 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=deleted 2024-09-12T06:53:28.024 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_image_id 2024-09-12T06:53:28.024 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_3 2024-09-12T06:53:28.024 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:53:28.024 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:53:28.024 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_3 2024-09-12T06:53:28.024 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-12T06:53:28.025 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:53:28.283 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_image_id=10a8f2f052cd 2024-09-12T06:53:28.284 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 10a8f2f052cd 2024-09-12T06:53:28.284 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -z 10a8f2f052cd -o 10a8f2f052cd = 10a8f2f052cd 2024-09-12T06:53:28.284 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_state=present 2024-09-12T06:53:28.284 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test present = present 2024-09-12T06:53:28.284 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:53:28.284 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ disable_mirror cluster2 mirror image_3 2024-09-12T06:53:28.284 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:53:28.284 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:53:28.284 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_3 2024-09-12T06:53:28.284 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_3 2024-09-12T06:53:56.802 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:Mirroring disabled 2024-09-12T06:53:56.809 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_present cluster1 mirror image_3 deleted 2024-09-12T06:53:56.810 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:53:56.810 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:53:56.810 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_3 2024-09-12T06:53:56.810 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=deleted 2024-09-12T06:53:56.810 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:53:56.810 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:53:56.810 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:53:56.810 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_3 2024-09-12T06:53:56.810 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:53:56.810 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:53:56.810 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_3 2024-09-12T06:53:56.810 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-12T06:53:56.811 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:53:56.848 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-09-12T06:53:56.851 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image_id= 2024-09-12T06:53:56.851 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T06:53:56.851 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.1 2024-09-12T06:53:56.953 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_present cluster1 mirror image_3 deleted '' 2024-09-12T06:53:56.953 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:53:56.953 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:53:56.953 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_3 2024-09-12T06:53:56.953 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=deleted 2024-09-12T06:53:56.953 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:53:56.953 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=deleted 2024-09-12T06:53:56.953 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_image_id 2024-09-12T06:53:56.953 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_3 2024-09-12T06:53:56.953 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:53:56.954 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:53:56.954 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_3 2024-09-12T06:53:56.954 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-12T06:53:56.954 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:53:56.987 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-09-12T06:53:56.992 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_image_id= 2024-09-12T06:53:56.992 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:53:56.992 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test deleted = deleted 2024-09-12T06:53:56.992 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:53:56.992 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ purge_snapshots cluster2 mirror image_3 2024-09-12T06:53:56.992 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:53:56.992 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:53:56.992 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_3 2024-09-12T06:53:56.992 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 snap purge mirror/image_3 2024-09-12T06:53:57.038 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image_retry cluster2 mirror image_3 2024-09-12T06:53:57.038 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:53:57.038 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:53:57.039 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_3 2024-09-12T06:53:57.039 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T06:53:57.039 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:53:57.039 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image cluster2 mirror image_3 2024-09-12T06:53:57.040 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:53:57.040 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:53:57.040 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_3 2024-09-12T06:53:57.040 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2024-09-12T06:53:57.086 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2024-09-12T06:53:57.295 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-12T06:53:57.299 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:53:57.299 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 10` 2024-09-12T06:53:57.299 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_4 2024-09-12T06:53:57.299 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image cluster2 mirror image_4 512M 2024-09-12T06:53:57.299 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:53:57.299 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:53:57.299 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:53:57.299 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:53:57.299 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_4 2024-09-12T06:53:57.300 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:53:57.300 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local size=128 2024-09-12T06:53:57.300 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n 512M ']' 2024-09-12T06:53:57.300 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ size=512M 2024-09-12T06:53:57.300 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:53:57.300 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/image_4 2024-09-12T06:53:57.357 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ enable_mirror cluster2 mirror image_4 2024-09-12T06:53:57.358 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:53:57.358 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:53:57.358 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_4 2024-09-12T06:53:57.358 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local mode=journal 2024-09-12T06:53:57.358 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_4 journal 2024-09-12T06:53:57.407 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:Mirroring enabled 2024-09-12T06:53:57.413 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 info mirror/image_4 2024-09-12T06:53:57.454 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:rbd image 'image_4': 2024-09-12T06:53:57.454 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: size 512 MiB in 128 objects 2024-09-12T06:53:57.454 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: order 22 (4 MiB objects) 2024-09-12T06:53:57.454 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: snapshot_count: 0 2024-09-12T06:53:57.454 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: id: 19dde0c0ddc1 2024-09-12T06:53:57.454 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: block_name_prefix: rbd_data.19dde0c0ddc1 2024-09-12T06:53:57.454 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: format: 2 2024-09-12T06:53:57.454 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: features: layering, exclusive-lock, journaling 2024-09-12T06:53:57.454 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: op_features: 2024-09-12T06:53:57.454 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: flags: 2024-09-12T06:53:57.454 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: create_timestamp: Thu Sep 12 06:53:57 2024 2024-09-12T06:53:57.454 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: access_timestamp: Thu Sep 12 06:53:57 2024 2024-09-12T06:53:57.454 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: modify_timestamp: Thu Sep 12 06:53:57 2024 2024-09-12T06:53:57.455 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: journal: 19dde0c0ddc1 2024-09-12T06:53:57.455 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring state: enabled 2024-09-12T06:53:57.455 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring mode: journal 2024-09-12T06:53:57.455 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring global id: f6e5921d-70b8-4a75-9d9b-dc63a0b39b33 2024-09-12T06:53:57.455 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring primary: true 2024-09-12T06:53:57.457 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ stress_write_image cluster2 mirror image_4 2024-09-12T06:53:57.457 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:53:57.457 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:53:57.457 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_4 2024-09-12T06:53:57.457 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T06:53:57.459 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local duration=12 2024-09-12T06:53:57.459 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set +e 2024-09-12T06:53:57.459 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_4 --debug-rbd=20 --debug-journaler=20 2024-09-12T06:53:58.036 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:100 IOs, 6351872 bytes 2024-09-12T06:54:03.103 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:200 IOs, 83405824 bytes 2024-09-12T06:54:05.562 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:300 IOs, 140782592 bytes 2024-09-12T06:54:06.091 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:400 IOs, 146113536 bytes 2024-09-12T06:54:09.097 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:500 IOs, 202330112 bytes 2024-09-12T06:54:09.470 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ error_code=124 2024-09-12T06:54:09.470 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set -e 2024-09-12T06:54:09.470 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T06:54:09.470 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:54:09.470 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_present cluster1 mirror image_4 present 2024-09-12T06:54:09.470 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:54:09.471 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:54:09.471 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_4 2024-09-12T06:54:09.471 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=present 2024-09-12T06:54:09.471 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:54:09.471 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:54:09.471 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:54:09.471 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_4 2024-09-12T06:54:09.471 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:54:09.471 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:54:09.471 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_4 2024-09-12T06:54:09.471 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-12T06:54:09.472 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:54:09.521 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image_id=10afaded9bfa 2024-09-12T06:54:09.521 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T06:54:09.521 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.1 2024-09-12T06:54:09.623 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_present cluster1 mirror image_4 present 10afaded9bfa 2024-09-12T06:54:09.623 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:54:09.623 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:54:09.623 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_4 2024-09-12T06:54:09.623 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=present 2024-09-12T06:54:09.623 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id=10afaded9bfa 2024-09-12T06:54:09.623 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=deleted 2024-09-12T06:54:09.623 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_image_id 2024-09-12T06:54:09.623 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_4 2024-09-12T06:54:09.623 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:54:09.624 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:54:09.624 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_4 2024-09-12T06:54:09.624 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-12T06:54:09.624 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:54:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_image_id=10afaded9bfa 2024-09-12T06:54:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 10afaded9bfa 2024-09-12T06:54:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -z 10afaded9bfa -o 10afaded9bfa = 10afaded9bfa 2024-09-12T06:54:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_state=present 2024-09-12T06:54:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test present = present 2024-09-12T06:54:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:54:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ disable_mirror cluster2 mirror image_4 2024-09-12T06:54:09.671 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:54:09.671 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:54:09.671 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_4 2024-09-12T06:54:09.671 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_4 2024-09-12T06:54:40.210 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:Mirroring disabled 2024-09-12T06:54:40.218 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_present cluster1 mirror image_4 deleted 2024-09-12T06:54:40.219 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:54:40.219 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:54:40.219 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_4 2024-09-12T06:54:40.219 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=deleted 2024-09-12T06:54:40.219 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:54:40.219 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:54:40.219 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:54:40.219 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_4 2024-09-12T06:54:40.219 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:54:40.219 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:54:40.219 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_4 2024-09-12T06:54:40.219 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-12T06:54:40.219 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:54:40.254 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-09-12T06:54:40.257 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image_id= 2024-09-12T06:54:40.257 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T06:54:40.257 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.1 2024-09-12T06:54:40.359 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_present cluster1 mirror image_4 deleted '' 2024-09-12T06:54:40.360 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:54:40.360 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:54:40.360 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_4 2024-09-12T06:54:40.360 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=deleted 2024-09-12T06:54:40.360 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:54:40.360 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=deleted 2024-09-12T06:54:40.360 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_image_id 2024-09-12T06:54:40.360 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_4 2024-09-12T06:54:40.360 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:54:40.360 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:54:40.360 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_4 2024-09-12T06:54:40.360 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-12T06:54:40.360 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:54:40.394 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-09-12T06:54:40.399 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_image_id= 2024-09-12T06:54:40.399 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:54:40.399 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test deleted = deleted 2024-09-12T06:54:40.399 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:54:40.399 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ purge_snapshots cluster2 mirror image_4 2024-09-12T06:54:40.399 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:54:40.399 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:54:40.399 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_4 2024-09-12T06:54:40.400 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 snap purge mirror/image_4 2024-09-12T06:54:40.445 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image_retry cluster2 mirror image_4 2024-09-12T06:54:40.445 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:54:40.446 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:54:40.446 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_4 2024-09-12T06:54:40.446 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T06:54:40.446 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:54:40.447 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image cluster2 mirror image_4 2024-09-12T06:54:40.447 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:54:40.447 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:54:40.447 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_4 2024-09-12T06:54:40.447 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2024-09-12T06:54:40.492 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2024-09-12T06:54:40.705 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-12T06:54:40.709 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:54:40.709 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 10` 2024-09-12T06:54:40.710 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_5 2024-09-12T06:54:40.710 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image cluster2 mirror image_5 512M 2024-09-12T06:54:40.710 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:54:40.710 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:54:40.710 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:54:40.710 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:54:40.710 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_5 2024-09-12T06:54:40.710 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:54:40.710 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local size=128 2024-09-12T06:54:40.710 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n 512M ']' 2024-09-12T06:54:40.710 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ size=512M 2024-09-12T06:54:40.710 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:54:40.710 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/image_5 2024-09-12T06:54:40.767 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ enable_mirror cluster2 mirror image_5 2024-09-12T06:54:40.767 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:54:40.767 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:54:40.767 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_5 2024-09-12T06:54:40.768 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local mode=journal 2024-09-12T06:54:40.768 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_5 journal 2024-09-12T06:54:40.816 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:Mirroring enabled 2024-09-12T06:54:40.826 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 info mirror/image_5 2024-09-12T06:54:40.865 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:rbd image 'image_5': 2024-09-12T06:54:40.866 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: size 512 MiB in 128 objects 2024-09-12T06:54:40.866 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: order 22 (4 MiB objects) 2024-09-12T06:54:40.866 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: snapshot_count: 0 2024-09-12T06:54:40.866 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: id: 19f5e5f5d361 2024-09-12T06:54:40.866 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: block_name_prefix: rbd_data.19f5e5f5d361 2024-09-12T06:54:40.866 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: format: 2 2024-09-12T06:54:40.866 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: features: layering, exclusive-lock, journaling 2024-09-12T06:54:40.866 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: op_features: 2024-09-12T06:54:40.866 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: flags: 2024-09-12T06:54:40.866 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: create_timestamp: Thu Sep 12 06:54:40 2024 2024-09-12T06:54:40.866 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: access_timestamp: Thu Sep 12 06:54:40 2024 2024-09-12T06:54:40.867 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: modify_timestamp: Thu Sep 12 06:54:40 2024 2024-09-12T06:54:40.867 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: journal: 19f5e5f5d361 2024-09-12T06:54:40.867 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring state: enabled 2024-09-12T06:54:40.867 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring mode: journal 2024-09-12T06:54:40.867 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring global id: ed8e7f22-e043-4b6c-b31d-1d4d37a956dd 2024-09-12T06:54:40.867 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring primary: true 2024-09-12T06:54:40.869 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ stress_write_image cluster2 mirror image_5 2024-09-12T06:54:40.869 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:54:40.869 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:54:40.869 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_5 2024-09-12T06:54:40.869 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T06:54:40.871 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local duration=8 2024-09-12T06:54:40.871 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set +e 2024-09-12T06:54:40.871 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_5 --debug-rbd=20 --debug-journaler=20 2024-09-12T06:54:42.140 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:100 IOs, 20843520 bytes 2024-09-12T06:54:45.561 INFO:tasks.ceph.cluster2.osd.2.smithi152.stderr:2024-09-12T06:54:45.561+0000 7f4789beb640 -1 reset not still connected to 0x56163e3b24e0 2024-09-12T06:54:45.561 INFO:tasks.ceph.cluster2.osd.2.smithi152.stderr:2024-09-12T06:54:45.561+0000 7f4789beb640 -1 reset not still connected to 0x56163e3b3c70 2024-09-12T06:54:45.562 INFO:tasks.ceph.cluster2.osd.2.smithi152.stderr:2024-09-12T06:54:45.561+0000 7f4789beb640 -1 reset not still connected to 0x56163f9b6340 2024-09-12T06:54:45.655 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:200 IOs, 90092544 bytes 2024-09-12T06:54:47.060 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:300 IOs, 92081152 bytes 2024-09-12T06:54:48.870 INFO:tasks.ceph.cluster2.osd.0.smithi152.stderr:2024-09-12T06:54:48.872+0000 7fd605c6c640 -1 bdev(0x55e762622c00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2024-09-12T06:54:48.882 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ error_code=124 2024-09-12T06:54:48.882 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set -e 2024-09-12T06:54:48.882 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T06:54:48.882 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:54:48.883 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_present cluster1 mirror image_5 present 2024-09-12T06:54:48.883 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:54:48.883 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:54:48.883 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_5 2024-09-12T06:54:48.883 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=present 2024-09-12T06:54:48.883 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:54:48.883 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:54:48.883 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:54:48.883 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_5 2024-09-12T06:54:48.883 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:54:48.883 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:54:48.883 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_5 2024-09-12T06:54:48.884 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-12T06:54:48.884 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:54:48.962 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image_id=10a8c960a005 2024-09-12T06:54:48.962 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T06:54:48.962 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.1 2024-09-12T06:54:49.064 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_present cluster1 mirror image_5 present 10a8c960a005 2024-09-12T06:54:49.064 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:54:49.064 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:54:49.064 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_5 2024-09-12T06:54:49.064 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=present 2024-09-12T06:54:49.064 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id=10a8c960a005 2024-09-12T06:54:49.064 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=deleted 2024-09-12T06:54:49.064 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_image_id 2024-09-12T06:54:49.064 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_5 2024-09-12T06:54:49.064 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:54:49.064 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:54:49.064 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_5 2024-09-12T06:54:49.065 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-12T06:54:49.065 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:54:49.112 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_image_id=10a8c960a005 2024-09-12T06:54:49.112 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 10a8c960a005 2024-09-12T06:54:49.112 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -z 10a8c960a005 -o 10a8c960a005 = 10a8c960a005 2024-09-12T06:54:49.113 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_state=present 2024-09-12T06:54:49.113 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test present = present 2024-09-12T06:54:49.113 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:54:49.113 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ disable_mirror cluster2 mirror image_5 2024-09-12T06:54:49.113 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:54:49.113 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:54:49.113 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_5 2024-09-12T06:54:49.113 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_5 2024-09-12T06:54:49.170 INFO:tasks.ceph.cluster2.osd.2.smithi152.stderr:2024-09-12T06:54:49.172+0000 7f4789beb640 -1 reset not still connected to 0x56163d5196c0 2024-09-12T06:54:49.171 INFO:tasks.ceph.cluster2.osd.2.smithi152.stderr:2024-09-12T06:54:49.172+0000 7f4789beb640 -1 reset not still connected to 0x56163d519e10 2024-09-12T06:54:49.171 INFO:tasks.ceph.cluster2.osd.2.smithi152.stderr:2024-09-12T06:54:49.172+0000 7f4789beb640 -1 reset not still connected to 0x56163df360d0 2024-09-12T06:54:49.171 INFO:tasks.ceph.cluster2.osd.2.smithi152.stderr:2024-09-12T06:54:49.172+0000 7f4789beb640 -1 reset not still connected to 0x56163df36270 2024-09-12T06:55:18.183 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:Mirroring disabled 2024-09-12T06:55:18.193 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_present cluster1 mirror image_5 deleted 2024-09-12T06:55:18.193 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:55:18.193 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:55:18.193 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_5 2024-09-12T06:55:18.193 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=deleted 2024-09-12T06:55:18.194 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:55:18.194 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:55:18.194 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:55:18.194 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_5 2024-09-12T06:55:18.194 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:55:18.194 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:55:18.194 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_5 2024-09-12T06:55:18.194 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-12T06:55:18.194 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:55:18.230 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-09-12T06:55:18.233 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image_id= 2024-09-12T06:55:18.233 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T06:55:18.233 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.1 2024-09-12T06:55:18.335 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_present cluster1 mirror image_5 deleted '' 2024-09-12T06:55:18.335 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:55:18.335 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:55:18.335 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_5 2024-09-12T06:55:18.335 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=deleted 2024-09-12T06:55:18.335 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:55:18.335 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=deleted 2024-09-12T06:55:18.335 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_image_id 2024-09-12T06:55:18.335 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_5 2024-09-12T06:55:18.335 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:55:18.336 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:55:18.336 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_5 2024-09-12T06:55:18.336 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-12T06:55:18.336 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:55:18.370 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-09-12T06:55:18.374 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_image_id= 2024-09-12T06:55:18.374 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:55:18.374 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test deleted = deleted 2024-09-12T06:55:18.374 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:55:18.375 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ purge_snapshots cluster2 mirror image_5 2024-09-12T06:55:18.375 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:55:18.375 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:55:18.375 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_5 2024-09-12T06:55:18.375 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 snap purge mirror/image_5 2024-09-12T06:55:18.421 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image_retry cluster2 mirror image_5 2024-09-12T06:55:18.421 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:55:18.421 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:55:18.421 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_5 2024-09-12T06:55:18.421 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T06:55:18.421 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:55:18.422 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image cluster2 mirror image_5 2024-09-12T06:55:18.422 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:55:18.422 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:55:18.422 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_5 2024-09-12T06:55:18.422 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2024-09-12T06:55:18.467 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2024-09-12T06:55:18.690 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-12T06:55:18.694 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:55:18.694 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 10` 2024-09-12T06:55:18.695 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_6 2024-09-12T06:55:18.695 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image cluster2 mirror image_6 512M 2024-09-12T06:55:18.695 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:55:18.695 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:55:18.695 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:55:18.695 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:55:18.695 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_6 2024-09-12T06:55:18.695 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:55:18.695 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local size=128 2024-09-12T06:55:18.695 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n 512M ']' 2024-09-12T06:55:18.695 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ size=512M 2024-09-12T06:55:18.695 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:55:18.696 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/image_6 2024-09-12T06:55:18.754 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ enable_mirror cluster2 mirror image_6 2024-09-12T06:55:18.754 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:55:18.754 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:55:18.754 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_6 2024-09-12T06:55:18.754 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local mode=journal 2024-09-12T06:55:18.754 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_6 journal 2024-09-12T06:55:18.804 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:Mirroring enabled 2024-09-12T06:55:18.809 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 info mirror/image_6 2024-09-12T06:55:18.849 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:rbd image 'image_6': 2024-09-12T06:55:18.849 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: size 512 MiB in 128 objects 2024-09-12T06:55:18.850 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: order 22 (4 MiB objects) 2024-09-12T06:55:18.850 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: snapshot_count: 0 2024-09-12T06:55:18.850 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: id: 1a0d9b861a62 2024-09-12T06:55:18.850 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: block_name_prefix: rbd_data.1a0d9b861a62 2024-09-12T06:55:18.850 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: format: 2 2024-09-12T06:55:18.850 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: features: layering, exclusive-lock, journaling 2024-09-12T06:55:18.850 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: op_features: 2024-09-12T06:55:18.850 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: flags: 2024-09-12T06:55:18.850 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: create_timestamp: Thu Sep 12 06:55:18 2024 2024-09-12T06:55:18.851 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: access_timestamp: Thu Sep 12 06:55:18 2024 2024-09-12T06:55:18.851 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: modify_timestamp: Thu Sep 12 06:55:18 2024 2024-09-12T06:55:18.851 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: journal: 1a0d9b861a62 2024-09-12T06:55:18.851 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring state: enabled 2024-09-12T06:55:18.851 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring mode: journal 2024-09-12T06:55:18.851 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring global id: dce12405-c10d-4f0f-95b3-f21718f87c9b 2024-09-12T06:55:18.851 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring primary: true 2024-09-12T06:55:18.853 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ stress_write_image cluster2 mirror image_6 2024-09-12T06:55:18.853 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:55:18.854 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:55:18.854 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_6 2024-09-12T06:55:18.854 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T06:55:18.854 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local duration=13 2024-09-12T06:55:18.854 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set +e 2024-09-12T06:55:18.854 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_6 --debug-rbd=20 --debug-journaler=20 2024-09-12T06:55:23.397 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:100 IOs, 76889088 bytes 2024-09-12T06:55:24.689 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:200 IOs, 95675392 bytes 2024-09-12T06:55:28.276 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:300 IOs, 183767040 bytes 2024-09-12T06:55:28.614 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:400 IOs, 185500672 bytes 2024-09-12T06:55:30.608 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:500 IOs, 239399936 bytes 2024-09-12T06:55:31.865 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ error_code=124 2024-09-12T06:55:31.865 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set -e 2024-09-12T06:55:31.865 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T06:55:31.865 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:55:31.865 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_present cluster1 mirror image_6 present 2024-09-12T06:55:31.865 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:55:31.865 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:55:31.865 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_6 2024-09-12T06:55:31.865 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=present 2024-09-12T06:55:31.865 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:55:31.865 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:55:31.866 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:55:31.866 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_6 2024-09-12T06:55:31.866 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:55:31.866 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:55:31.866 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_6 2024-09-12T06:55:31.867 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-12T06:55:31.868 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:55:31.915 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image_id=10a860dd4a17 2024-09-12T06:55:31.916 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T06:55:31.916 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.1 2024-09-12T06:55:32.017 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_present cluster1 mirror image_6 present 10a860dd4a17 2024-09-12T06:55:32.018 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:55:32.018 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:55:32.018 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_6 2024-09-12T06:55:32.018 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=present 2024-09-12T06:55:32.018 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id=10a860dd4a17 2024-09-12T06:55:32.018 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=deleted 2024-09-12T06:55:32.018 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_image_id 2024-09-12T06:55:32.018 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_6 2024-09-12T06:55:32.018 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:55:32.018 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:55:32.018 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_6 2024-09-12T06:55:32.018 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-12T06:55:32.018 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:55:32.064 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_image_id=10a860dd4a17 2024-09-12T06:55:32.065 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 10a860dd4a17 2024-09-12T06:55:32.065 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -z 10a860dd4a17 -o 10a860dd4a17 = 10a860dd4a17 2024-09-12T06:55:32.065 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_state=present 2024-09-12T06:55:32.065 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test present = present 2024-09-12T06:55:32.065 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:55:32.065 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ disable_mirror cluster2 mirror image_6 2024-09-12T06:55:32.065 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:55:32.065 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:55:32.065 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_6 2024-09-12T06:55:32.065 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_6 2024-09-12T06:56:01.240 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:Mirroring disabled 2024-09-12T06:56:01.248 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_present cluster1 mirror image_6 deleted 2024-09-12T06:56:01.248 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:56:01.248 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:56:01.248 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_6 2024-09-12T06:56:01.248 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=deleted 2024-09-12T06:56:01.249 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:56:01.249 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:56:01.249 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:56:01.249 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_6 2024-09-12T06:56:01.249 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:56:01.249 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:56:01.249 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_6 2024-09-12T06:56:01.249 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-12T06:56:01.249 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:56:01.284 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-09-12T06:56:01.287 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image_id= 2024-09-12T06:56:01.287 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T06:56:01.287 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.1 2024-09-12T06:56:01.389 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_present cluster1 mirror image_6 deleted '' 2024-09-12T06:56:01.389 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:56:01.389 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:56:01.389 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_6 2024-09-12T06:56:01.389 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=deleted 2024-09-12T06:56:01.390 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:56:01.390 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=deleted 2024-09-12T06:56:01.390 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_image_id 2024-09-12T06:56:01.390 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_6 2024-09-12T06:56:01.390 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:56:01.390 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:56:01.390 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_6 2024-09-12T06:56:01.390 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-12T06:56:01.390 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:56:01.426 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-09-12T06:56:01.430 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_image_id= 2024-09-12T06:56:01.430 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:56:01.430 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test deleted = deleted 2024-09-12T06:56:01.430 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:56:01.430 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ purge_snapshots cluster2 mirror image_6 2024-09-12T06:56:01.430 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:56:01.430 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:56:01.431 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_6 2024-09-12T06:56:01.431 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 snap purge mirror/image_6 2024-09-12T06:56:01.477 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image_retry cluster2 mirror image_6 2024-09-12T06:56:01.477 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:56:01.478 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:56:01.478 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_6 2024-09-12T06:56:01.478 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T06:56:01.478 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:56:01.478 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image cluster2 mirror image_6 2024-09-12T06:56:01.479 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:56:01.479 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:56:01.479 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_6 2024-09-12T06:56:01.479 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2024-09-12T06:56:01.525 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2024-09-12T06:56:01.741 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-12T06:56:01.745 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:56:01.745 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 10` 2024-09-12T06:56:01.745 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_7 2024-09-12T06:56:01.745 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image cluster2 mirror image_7 512M 2024-09-12T06:56:01.745 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:56:01.745 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:56:01.745 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:56:01.745 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:56:01.745 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_7 2024-09-12T06:56:01.746 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:56:01.746 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local size=128 2024-09-12T06:56:01.746 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n 512M ']' 2024-09-12T06:56:01.746 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ size=512M 2024-09-12T06:56:01.746 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:56:01.746 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/image_7 2024-09-12T06:56:01.804 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ enable_mirror cluster2 mirror image_7 2024-09-12T06:56:01.804 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:56:01.804 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:56:01.804 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_7 2024-09-12T06:56:01.804 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local mode=journal 2024-09-12T06:56:01.804 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_7 journal 2024-09-12T06:56:01.853 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:Mirroring enabled 2024-09-12T06:56:01.859 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 info mirror/image_7 2024-09-12T06:56:01.898 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:rbd image 'image_7': 2024-09-12T06:56:01.899 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: size 512 MiB in 128 objects 2024-09-12T06:56:01.899 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: order 22 (4 MiB objects) 2024-09-12T06:56:01.899 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: snapshot_count: 0 2024-09-12T06:56:01.899 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: id: 1a25b355603b 2024-09-12T06:56:01.899 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: block_name_prefix: rbd_data.1a25b355603b 2024-09-12T06:56:01.899 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: format: 2 2024-09-12T06:56:01.899 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: features: layering, exclusive-lock, journaling 2024-09-12T06:56:01.899 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: op_features: 2024-09-12T06:56:01.899 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: flags: 2024-09-12T06:56:01.899 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: create_timestamp: Thu Sep 12 06:56:01 2024 2024-09-12T06:56:01.899 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: access_timestamp: Thu Sep 12 06:56:01 2024 2024-09-12T06:56:01.899 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: modify_timestamp: Thu Sep 12 06:56:01 2024 2024-09-12T06:56:01.899 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: journal: 1a25b355603b 2024-09-12T06:56:01.900 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring state: enabled 2024-09-12T06:56:01.900 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring mode: journal 2024-09-12T06:56:01.900 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring global id: 5694de24-1da1-4541-a409-63f1be4deffc 2024-09-12T06:56:01.900 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring primary: true 2024-09-12T06:56:01.902 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ stress_write_image cluster2 mirror image_7 2024-09-12T06:56:01.902 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:56:01.902 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:56:01.902 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_7 2024-09-12T06:56:01.902 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T06:56:01.904 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local duration=5 2024-09-12T06:56:01.904 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set +e 2024-09-12T06:56:01.904 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_7 --debug-rbd=20 --debug-journaler=20 2024-09-12T06:56:04.752 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:100 IOs, 77284352 bytes 2024-09-12T06:56:06.914 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ error_code=124 2024-09-12T06:56:06.914 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set -e 2024-09-12T06:56:06.914 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T06:56:06.914 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:56:06.914 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_present cluster1 mirror image_7 present 2024-09-12T06:56:06.914 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:56:06.914 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:56:06.915 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_7 2024-09-12T06:56:06.915 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=present 2024-09-12T06:56:06.915 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:56:06.915 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:56:06.915 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:56:06.916 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_7 2024-09-12T06:56:06.916 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:56:06.916 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:56:06.916 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_7 2024-09-12T06:56:06.917 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:56:06.917 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-12T06:56:06.985 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image_id=10a8e28634ac 2024-09-12T06:56:06.986 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T06:56:06.986 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.1 2024-09-12T06:56:07.087 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_present cluster1 mirror image_7 present 10a8e28634ac 2024-09-12T06:56:07.088 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:56:07.088 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:56:07.088 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_7 2024-09-12T06:56:07.088 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=present 2024-09-12T06:56:07.088 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id=10a8e28634ac 2024-09-12T06:56:07.088 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=deleted 2024-09-12T06:56:07.088 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_image_id 2024-09-12T06:56:07.088 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_7 2024-09-12T06:56:07.089 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:56:07.089 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:56:07.089 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_7 2024-09-12T06:56:07.089 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-12T06:56:07.089 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:56:07.133 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_image_id=10a8e28634ac 2024-09-12T06:56:07.133 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 10a8e28634ac 2024-09-12T06:56:07.133 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -z 10a8e28634ac -o 10a8e28634ac = 10a8e28634ac 2024-09-12T06:56:07.133 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_state=present 2024-09-12T06:56:07.133 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test present = present 2024-09-12T06:56:07.134 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:56:07.134 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ disable_mirror cluster2 mirror image_7 2024-09-12T06:56:07.134 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:56:07.134 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:56:07.134 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_7 2024-09-12T06:56:07.134 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_7 2024-09-12T06:56:39.865 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:Mirroring disabled 2024-09-12T06:56:39.873 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_present cluster1 mirror image_7 deleted 2024-09-12T06:56:39.873 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:56:39.873 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:56:39.873 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_7 2024-09-12T06:56:39.873 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=deleted 2024-09-12T06:56:39.873 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:56:39.874 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:56:39.874 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:56:39.874 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_7 2024-09-12T06:56:39.874 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:56:39.874 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:56:39.874 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_7 2024-09-12T06:56:39.874 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-12T06:56:39.875 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:56:39.908 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-09-12T06:56:39.912 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image_id= 2024-09-12T06:56:39.912 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T06:56:39.912 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.1 2024-09-12T06:56:40.013 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_present cluster1 mirror image_7 deleted '' 2024-09-12T06:56:40.013 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:56:40.014 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:56:40.014 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_7 2024-09-12T06:56:40.014 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=deleted 2024-09-12T06:56:40.014 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:56:40.014 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=deleted 2024-09-12T06:56:40.014 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_image_id 2024-09-12T06:56:40.014 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_7 2024-09-12T06:56:40.014 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:56:40.014 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:56:40.014 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_7 2024-09-12T06:56:40.014 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-12T06:56:40.014 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:56:40.051 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-09-12T06:56:40.055 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_image_id= 2024-09-12T06:56:40.055 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:56:40.056 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test deleted = deleted 2024-09-12T06:56:40.056 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:56:40.056 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ purge_snapshots cluster2 mirror image_7 2024-09-12T06:56:40.056 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:56:40.056 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:56:40.056 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_7 2024-09-12T06:56:40.056 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 snap purge mirror/image_7 2024-09-12T06:56:40.103 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image_retry cluster2 mirror image_7 2024-09-12T06:56:40.103 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:56:40.103 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:56:40.104 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_7 2024-09-12T06:56:40.104 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T06:56:40.104 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:56:40.105 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image cluster2 mirror image_7 2024-09-12T06:56:40.105 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:56:40.105 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:56:40.105 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_7 2024-09-12T06:56:40.106 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2024-09-12T06:56:40.153 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2024-09-12T06:56:40.361 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-12T06:56:40.365 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:56:40.365 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 10` 2024-09-12T06:56:40.365 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_8 2024-09-12T06:56:40.365 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image cluster2 mirror image_8 512M 2024-09-12T06:56:40.365 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:56:40.365 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:56:40.365 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:56:40.366 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:56:40.366 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_8 2024-09-12T06:56:40.366 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:56:40.366 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local size=128 2024-09-12T06:56:40.366 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n 512M ']' 2024-09-12T06:56:40.366 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ size=512M 2024-09-12T06:56:40.366 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:56:40.366 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/image_8 2024-09-12T06:56:40.423 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ enable_mirror cluster2 mirror image_8 2024-09-12T06:56:40.424 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:56:40.424 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:56:40.424 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_8 2024-09-12T06:56:40.424 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local mode=journal 2024-09-12T06:56:40.424 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_8 journal 2024-09-12T06:56:40.471 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:Mirroring enabled 2024-09-12T06:56:40.477 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 info mirror/image_8 2024-09-12T06:56:40.517 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:rbd image 'image_8': 2024-09-12T06:56:40.517 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: size 512 MiB in 128 objects 2024-09-12T06:56:40.517 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: order 22 (4 MiB objects) 2024-09-12T06:56:40.517 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: snapshot_count: 0 2024-09-12T06:56:40.517 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: id: 1a3d6a3f0260 2024-09-12T06:56:40.517 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: block_name_prefix: rbd_data.1a3d6a3f0260 2024-09-12T06:56:40.518 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: format: 2 2024-09-12T06:56:40.518 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: features: layering, exclusive-lock, journaling 2024-09-12T06:56:40.518 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: op_features: 2024-09-12T06:56:40.518 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: flags: 2024-09-12T06:56:40.518 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: create_timestamp: Thu Sep 12 06:56:40 2024 2024-09-12T06:56:40.518 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: access_timestamp: Thu Sep 12 06:56:40 2024 2024-09-12T06:56:40.518 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: modify_timestamp: Thu Sep 12 06:56:40 2024 2024-09-12T06:56:40.518 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: journal: 1a3d6a3f0260 2024-09-12T06:56:40.518 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring state: enabled 2024-09-12T06:56:40.518 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring mode: journal 2024-09-12T06:56:40.519 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring global id: b169e370-baef-4a29-9b2c-677da6e27cd6 2024-09-12T06:56:40.519 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring primary: true 2024-09-12T06:56:40.520 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ stress_write_image cluster2 mirror image_8 2024-09-12T06:56:40.520 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:56:40.520 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:56:40.520 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_8 2024-09-12T06:56:40.521 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T06:56:40.522 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local duration=10 2024-09-12T06:56:40.522 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set +e 2024-09-12T06:56:40.522 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_8 --debug-rbd=20 --debug-journaler=20 2024-09-12T06:56:43.877 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:100 IOs, 43091968 bytes 2024-09-12T06:56:46.362 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:200 IOs, 82283520 bytes 2024-09-12T06:56:47.968 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:300 IOs, 134166528 bytes 2024-09-12T06:56:50.533 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ error_code=124 2024-09-12T06:56:50.533 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set -e 2024-09-12T06:56:50.533 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T06:56:50.533 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:56:50.533 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_present cluster1 mirror image_8 present 2024-09-12T06:56:50.533 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:56:50.533 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:56:50.533 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_8 2024-09-12T06:56:50.533 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=present 2024-09-12T06:56:50.534 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:56:50.534 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:56:50.534 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:56:50.534 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_8 2024-09-12T06:56:50.534 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:56:50.534 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:56:50.534 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_8 2024-09-12T06:56:50.534 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-12T06:56:50.534 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:56:50.581 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image_id=10af93f1fd1f 2024-09-12T06:56:50.581 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T06:56:50.581 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.1 2024-09-12T06:56:50.682 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_present cluster1 mirror image_8 present 10af93f1fd1f 2024-09-12T06:56:50.682 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:56:50.683 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:56:50.683 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_8 2024-09-12T06:56:50.683 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=present 2024-09-12T06:56:50.683 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id=10af93f1fd1f 2024-09-12T06:56:50.683 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=deleted 2024-09-12T06:56:50.683 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_image_id 2024-09-12T06:56:50.683 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_8 2024-09-12T06:56:50.683 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:56:50.683 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:56:50.683 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_8 2024-09-12T06:56:50.683 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-12T06:56:50.684 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:56:50.730 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_image_id=10af93f1fd1f 2024-09-12T06:56:50.730 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 10af93f1fd1f 2024-09-12T06:56:50.731 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -z 10af93f1fd1f -o 10af93f1fd1f = 10af93f1fd1f 2024-09-12T06:56:50.731 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_state=present 2024-09-12T06:56:50.731 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test present = present 2024-09-12T06:56:50.731 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:56:50.731 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ disable_mirror cluster2 mirror image_8 2024-09-12T06:56:50.731 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:56:50.731 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:56:50.731 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_8 2024-09-12T06:56:50.731 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_8 2024-09-12T06:57:20.852 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:Mirroring disabled 2024-09-12T06:57:20.862 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_present cluster1 mirror image_8 deleted 2024-09-12T06:57:20.863 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:57:20.863 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:57:20.863 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_8 2024-09-12T06:57:20.863 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=deleted 2024-09-12T06:57:20.863 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:57:20.863 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:57:20.863 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:57:20.863 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_8 2024-09-12T06:57:20.863 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:57:20.863 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:57:20.864 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_8 2024-09-12T06:57:20.864 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-12T06:57:20.864 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:57:20.899 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-09-12T06:57:20.902 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image_id= 2024-09-12T06:57:20.902 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T06:57:20.902 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.1 2024-09-12T06:57:21.006 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_present cluster1 mirror image_8 deleted '' 2024-09-12T06:57:21.006 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:57:21.006 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:57:21.006 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_8 2024-09-12T06:57:21.006 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=deleted 2024-09-12T06:57:21.007 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:57:21.007 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=deleted 2024-09-12T06:57:21.007 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_image_id 2024-09-12T06:57:21.007 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_8 2024-09-12T06:57:21.007 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:57:21.008 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:57:21.008 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_8 2024-09-12T06:57:21.008 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-12T06:57:21.008 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:57:21.040 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-09-12T06:57:21.044 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_image_id= 2024-09-12T06:57:21.044 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:57:21.044 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test deleted = deleted 2024-09-12T06:57:21.044 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:57:21.044 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ purge_snapshots cluster2 mirror image_8 2024-09-12T06:57:21.044 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:57:21.044 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:57:21.044 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_8 2024-09-12T06:57:21.045 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 snap purge mirror/image_8 2024-09-12T06:57:21.091 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image_retry cluster2 mirror image_8 2024-09-12T06:57:21.092 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:57:21.092 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:57:21.092 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_8 2024-09-12T06:57:21.092 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T06:57:21.092 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:57:21.093 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image cluster2 mirror image_8 2024-09-12T06:57:21.093 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:57:21.093 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:57:21.093 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_8 2024-09-12T06:57:21.093 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2024-09-12T06:57:21.139 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2024-09-12T06:57:21.557 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-12T06:57:21.562 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:57:21.562 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 10` 2024-09-12T06:57:21.563 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_9 2024-09-12T06:57:21.563 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image cluster2 mirror image_9 512M 2024-09-12T06:57:21.563 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:57:21.563 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:57:21.563 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:57:21.563 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:57:21.564 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_9 2024-09-12T06:57:21.564 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:57:21.564 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local size=128 2024-09-12T06:57:21.564 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n 512M ']' 2024-09-12T06:57:21.564 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ size=512M 2024-09-12T06:57:21.565 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:57:21.565 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/image_9 2024-09-12T06:57:21.620 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ enable_mirror cluster2 mirror image_9 2024-09-12T06:57:21.620 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:57:21.621 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:57:21.621 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_9 2024-09-12T06:57:21.621 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local mode=journal 2024-09-12T06:57:21.621 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_9 journal 2024-09-12T06:57:21.668 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:Mirroring enabled 2024-09-12T06:57:21.674 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 info mirror/image_9 2024-09-12T06:57:21.714 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:rbd image 'image_9': 2024-09-12T06:57:21.715 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: size 512 MiB in 128 objects 2024-09-12T06:57:21.715 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: order 22 (4 MiB objects) 2024-09-12T06:57:21.715 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: snapshot_count: 0 2024-09-12T06:57:21.715 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: id: 1a554c8de739 2024-09-12T06:57:21.715 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: block_name_prefix: rbd_data.1a554c8de739 2024-09-12T06:57:21.715 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: format: 2 2024-09-12T06:57:21.715 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: features: layering, exclusive-lock, journaling 2024-09-12T06:57:21.715 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: op_features: 2024-09-12T06:57:21.716 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: flags: 2024-09-12T06:57:21.716 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: create_timestamp: Thu Sep 12 06:57:21 2024 2024-09-12T06:57:21.716 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: access_timestamp: Thu Sep 12 06:57:21 2024 2024-09-12T06:57:21.716 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: modify_timestamp: Thu Sep 12 06:57:21 2024 2024-09-12T06:57:21.716 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: journal: 1a554c8de739 2024-09-12T06:57:21.716 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring state: enabled 2024-09-12T06:57:21.716 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring mode: journal 2024-09-12T06:57:21.716 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring global id: 9dbcea53-b486-4e9e-b184-cd9d3ed0976e 2024-09-12T06:57:21.716 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring primary: true 2024-09-12T06:57:21.718 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ stress_write_image cluster2 mirror image_9 2024-09-12T06:57:21.718 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:57:21.718 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:57:21.718 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_9 2024-09-12T06:57:21.718 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T06:57:21.720 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local duration=12 2024-09-12T06:57:21.721 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set +e 2024-09-12T06:57:21.721 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_9 --debug-rbd=20 --debug-journaler=20 2024-09-12T06:57:22.634 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:100 IOs, 14294016 bytes 2024-09-12T06:57:22.932 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:200 IOs, 39578624 bytes 2024-09-12T06:57:25.935 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:300 IOs, 63437824 bytes 2024-09-12T06:57:26.895 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:400 IOs, 77614080 bytes 2024-09-12T06:57:28.771 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:500 IOs, 108082176 bytes 2024-09-12T06:57:30.233 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:600 IOs, 160043008 bytes 2024-09-12T06:57:33.730 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ error_code=124 2024-09-12T06:57:33.730 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set -e 2024-09-12T06:57:33.730 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T06:57:33.730 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:57:33.731 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_present cluster1 mirror image_9 present 2024-09-12T06:57:33.731 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:57:33.731 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:57:33.731 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_9 2024-09-12T06:57:33.731 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=present 2024-09-12T06:57:33.731 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:57:33.731 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:57:33.731 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:57:33.732 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_9 2024-09-12T06:57:33.732 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:57:33.732 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:57:33.732 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_9 2024-09-12T06:57:33.732 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-12T06:57:33.732 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:57:33.780 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image_id=10a89c9557ab 2024-09-12T06:57:33.780 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T06:57:33.780 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.1 2024-09-12T06:57:33.882 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_present cluster1 mirror image_9 present 10a89c9557ab 2024-09-12T06:57:33.882 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:57:33.882 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:57:33.882 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_9 2024-09-12T06:57:33.882 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=present 2024-09-12T06:57:33.882 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id=10a89c9557ab 2024-09-12T06:57:33.882 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=deleted 2024-09-12T06:57:33.882 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_image_id 2024-09-12T06:57:33.882 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_9 2024-09-12T06:57:33.882 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:57:33.882 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:57:33.882 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_9 2024-09-12T06:57:33.883 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-12T06:57:33.883 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:57:33.929 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_image_id=10a89c9557ab 2024-09-12T06:57:33.929 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 10a89c9557ab 2024-09-12T06:57:33.929 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -z 10a89c9557ab -o 10a89c9557ab = 10a89c9557ab 2024-09-12T06:57:33.929 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_state=present 2024-09-12T06:57:33.929 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test present = present 2024-09-12T06:57:33.929 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:57:33.929 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ disable_mirror cluster2 mirror image_9 2024-09-12T06:57:33.929 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:57:33.930 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:57:33.930 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_9 2024-09-12T06:57:33.930 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_9 2024-09-12T06:58:03.244 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:Mirroring disabled 2024-09-12T06:58:03.253 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_present cluster1 mirror image_9 deleted 2024-09-12T06:58:03.253 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:58:03.253 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:58:03.253 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_9 2024-09-12T06:58:03.253 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=deleted 2024-09-12T06:58:03.253 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:58:03.253 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:58:03.253 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:58:03.253 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_9 2024-09-12T06:58:03.254 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:58:03.254 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:58:03.254 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_9 2024-09-12T06:58:03.254 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-12T06:58:03.254 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:58:03.289 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-09-12T06:58:03.293 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image_id= 2024-09-12T06:58:03.293 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T06:58:03.293 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.1 2024-09-12T06:58:03.395 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_present cluster1 mirror image_9 deleted '' 2024-09-12T06:58:03.395 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:58:03.395 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:58:03.395 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_9 2024-09-12T06:58:03.395 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=deleted 2024-09-12T06:58:03.395 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:58:03.395 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=deleted 2024-09-12T06:58:03.396 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_image_id 2024-09-12T06:58:03.396 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_9 2024-09-12T06:58:03.396 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:58:03.396 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:58:03.396 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_9 2024-09-12T06:58:03.396 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-12T06:58:03.396 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:58:03.431 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-09-12T06:58:03.435 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_image_id= 2024-09-12T06:58:03.435 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:58:03.435 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test deleted = deleted 2024-09-12T06:58:03.435 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:58:03.435 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ purge_snapshots cluster2 mirror image_9 2024-09-12T06:58:03.435 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:58:03.435 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:58:03.435 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_9 2024-09-12T06:58:03.435 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 snap purge mirror/image_9 2024-09-12T06:58:03.482 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image_retry cluster2 mirror image_9 2024-09-12T06:58:03.482 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:58:03.482 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:58:03.482 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_9 2024-09-12T06:58:03.482 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T06:58:03.482 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:58:03.483 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image cluster2 mirror image_9 2024-09-12T06:58:03.483 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:58:03.483 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:58:03.483 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_9 2024-09-12T06:58:03.483 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2024-09-12T06:58:03.529 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2024-09-12T06:58:03.758 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-12T06:58:03.762 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:58:03.763 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for i in `seq 1 10` 2024-09-12T06:58:03.763 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image=image_10 2024-09-12T06:58:03.763 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ create_image cluster2 mirror image_10 512M 2024-09-12T06:58:03.763 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:58:03.763 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:58:03.763 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:58:03.763 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:58:03.763 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_10 2024-09-12T06:58:03.763 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:58:03.763 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local size=128 2024-09-12T06:58:03.763 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n 512M ']' 2024-09-12T06:58:03.763 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ size=512M 2024-09-12T06:58:03.764 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ shift 2024-09-12T06:58:03.764 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/image_10 2024-09-12T06:58:03.820 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ enable_mirror cluster2 mirror image_10 2024-09-12T06:58:03.820 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:58:03.820 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:58:03.820 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_10 2024-09-12T06:58:03.821 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local mode=journal 2024-09-12T06:58:03.821 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_10 journal 2024-09-12T06:58:03.869 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:Mirroring enabled 2024-09-12T06:58:03.875 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 info mirror/image_10 2024-09-12T06:58:03.915 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:rbd image 'image_10': 2024-09-12T06:58:03.915 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: size 512 MiB in 128 objects 2024-09-12T06:58:03.915 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: order 22 (4 MiB objects) 2024-09-12T06:58:03.915 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: snapshot_count: 0 2024-09-12T06:58:03.916 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: id: 1a6dffb89abb 2024-09-12T06:58:03.916 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: block_name_prefix: rbd_data.1a6dffb89abb 2024-09-12T06:58:03.916 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: format: 2 2024-09-12T06:58:03.916 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: features: layering, exclusive-lock, journaling 2024-09-12T06:58:03.916 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: op_features: 2024-09-12T06:58:03.916 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: flags: 2024-09-12T06:58:03.916 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: create_timestamp: Thu Sep 12 06:58:03 2024 2024-09-12T06:58:03.916 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: access_timestamp: Thu Sep 12 06:58:03 2024 2024-09-12T06:58:03.916 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: modify_timestamp: Thu Sep 12 06:58:03 2024 2024-09-12T06:58:03.916 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: journal: 1a6dffb89abb 2024-09-12T06:58:03.916 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring state: enabled 2024-09-12T06:58:03.916 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring mode: journal 2024-09-12T06:58:03.916 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring global id: e78ed032-ef5f-4c8e-87df-66bfb66a9fe4 2024-09-12T06:58:03.916 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout: mirroring primary: true 2024-09-12T06:58:03.919 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ stress_write_image cluster2 mirror image_10 2024-09-12T06:58:03.919 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:58:03.919 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:58:03.919 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_10 2024-09-12T06:58:03.919 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T06:58:03.921 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local duration=9 2024-09-12T06:58:03.921 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set +e 2024-09-12T06:58:03.921 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_10 --debug-rbd=20 --debug-journaler=20 2024-09-12T06:58:08.347 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:100 IOs, 92461056 bytes 2024-09-12T06:58:12.482 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:200 IOs, 148260864 bytes 2024-09-12T06:58:12.931 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ error_code=124 2024-09-12T06:58:12.931 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set -e 2024-09-12T06:58:12.931 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T06:58:12.931 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:58:12.931 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_present cluster1 mirror image_10 present 2024-09-12T06:58:12.931 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:58:12.931 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:58:12.931 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_10 2024-09-12T06:58:12.931 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=present 2024-09-12T06:58:12.931 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:58:12.932 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:58:12.932 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:58:12.932 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_10 2024-09-12T06:58:12.932 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:58:12.932 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:58:12.932 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_10 2024-09-12T06:58:12.932 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-12T06:58:12.932 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:58:12.983 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image_id=10a82906df9c 2024-09-12T06:58:12.983 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T06:58:12.983 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.1 2024-09-12T06:58:13.085 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_present cluster1 mirror image_10 present 10a82906df9c 2024-09-12T06:58:13.085 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:58:13.085 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:58:13.085 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_10 2024-09-12T06:58:13.085 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=present 2024-09-12T06:58:13.085 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id=10a82906df9c 2024-09-12T06:58:13.085 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=deleted 2024-09-12T06:58:13.086 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_image_id 2024-09-12T06:58:13.086 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_10 2024-09-12T06:58:13.086 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:58:13.086 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:58:13.086 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_10 2024-09-12T06:58:13.086 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-12T06:58:13.086 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:58:13.137 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_image_id=10a82906df9c 2024-09-12T06:58:13.137 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n 10a82906df9c 2024-09-12T06:58:13.137 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -z 10a82906df9c -o 10a82906df9c = 10a82906df9c 2024-09-12T06:58:13.137 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_state=present 2024-09-12T06:58:13.137 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test present = present 2024-09-12T06:58:13.138 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:58:13.138 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ disable_mirror cluster2 mirror image_10 2024-09-12T06:58:13.138 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:58:13.138 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:58:13.138 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_10 2024-09-12T06:58:13.138 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_10 2024-09-12T06:58:41.221 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:Mirroring disabled 2024-09-12T06:58:41.232 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_present cluster1 mirror image_10 deleted 2024-09-12T06:58:41.232 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:58:41.232 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:58:41.233 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_10 2024-09-12T06:58:41.233 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local state=deleted 2024-09-12T06:58:41.233 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:58:41.233 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local s 2024-09-12T06:58:41.233 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:58:41.233 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_10 2024-09-12T06:58:41.233 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:58:41.233 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:58:41.233 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_10 2024-09-12T06:58:41.233 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-12T06:58:41.233 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:58:41.269 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-09-12T06:58:41.272 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ image_id= 2024-09-12T06:58:41.272 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T06:58:41.272 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0.1 2024-09-12T06:58:41.374 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test_image_present cluster1 mirror image_10 deleted '' 2024-09-12T06:58:41.374 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:58:41.374 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:58:41.374 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_10 2024-09-12T06:58:41.374 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local test_state=deleted 2024-09-12T06:58:41.374 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image_id= 2024-09-12T06:58:41.374 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_state=deleted 2024-09-12T06:58:41.374 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local current_image_id 2024-09-12T06:58:41.374 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ get_image_id cluster1 mirror image_10 2024-09-12T06:58:41.375 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:58:41.375 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:58:41.375 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local image=image_10 2024-09-12T06:58:41.375 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-12T06:58:41.375 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T06:58:41.409 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-09-12T06:58:41.413 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ current_image_id= 2024-09-12T06:58:41.413 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:58:41.413 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test deleted = deleted 2024-09-12T06:58:41.413 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:58:41.414 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ purge_snapshots cluster2 mirror image_10 2024-09-12T06:58:41.414 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:58:41.414 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:58:41.414 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_10 2024-09-12T06:58:41.414 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster cluster2 snap purge mirror/image_10 2024-09-12T06:58:41.461 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image_retry cluster2 mirror image_10 2024-09-12T06:58:41.461 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:58:41.461 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:58:41.461 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_10 2024-09-12T06:58:41.461 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T06:58:41.461 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:58:41.462 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ remove_image cluster2 mirror image_10 2024-09-12T06:58:41.462 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:58:41.462 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:58:41.462 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local image=image_10 2024-09-12T06:58:41.462 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2024-09-12T06:58:41.510 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2024-09-12T06:58:41.719 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-12T06:58:41.722 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:58:41.722 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ testlog 'TEST: check if removed images'\'' OMAP are removed' 2024-09-12T06:58:41.723 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ tee -a /tmp/tmp.GzjHZGs9Z0/rbd-mirror.test.log 2024-09-12T06:58:41.723 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ date '+%F %T' 2024-09-12T06:58:41.724 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo 2024-09-12 06:58:41 TEST: check if removed 'images'\''' OMAP are removed 2024-09-12T06:58:41.724 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:2024-09-12 06:58:41 TEST: check if removed images' OMAP are removed 2024-09-12T06:58:41.725 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_in_omap cluster1 mirror 2024-09-12T06:58:41.725 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:58:41.725 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:58:41.725 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirroring status_global 2024-09-12T06:58:41.725 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:58:41.725 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:58:41.725 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local obj_name=rbd_mirroring 2024-09-12T06:58:41.725 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local filter=status_global 2024-09-12T06:58:41.725 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-12T06:58:41.725 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:58:41.726 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set +e 2024-09-12T06:58:41.727 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirroring status_global 2024-09-12T06:58:41.727 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:58:41.727 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:58:41.727 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local obj_name=rbd_mirroring 2024-09-12T06:58:41.727 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local filter=status_global 2024-09-12T06:58:41.727 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ list_omap_keys cluster1 mirror rbd_mirroring 2024-09-12T06:58:41.727 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:58:41.727 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:58:41.727 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local obj_name=rbd_mirroring 2024-09-12T06:58:41.727 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirroring 2024-09-12T06:58:41.727 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -c status_global 2024-09-12T06:58:41.856 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 0 = 0 2024-09-12T06:58:41.856 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ error_code=0 2024-09-12T06:58:41.856 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set -e 2024-09-12T06:58:41.856 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 -eq 0 ']' 2024-09-12T06:58:41.857 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:58:41.857 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirroring image_ 2024-09-12T06:58:41.857 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:58:41.857 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:58:41.857 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local obj_name=rbd_mirroring 2024-09-12T06:58:41.857 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local filter=image_ 2024-09-12T06:58:41.857 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-12T06:58:41.857 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:58:41.857 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set +e 2024-09-12T06:58:41.857 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirroring image_ 2024-09-12T06:58:41.857 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:58:41.858 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:58:41.858 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local obj_name=rbd_mirroring 2024-09-12T06:58:41.858 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local filter=image_ 2024-09-12T06:58:41.858 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ list_omap_keys cluster1 mirror rbd_mirroring 2024-09-12T06:58:41.858 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:58:41.858 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:58:41.858 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local obj_name=rbd_mirroring 2024-09-12T06:58:41.858 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirroring 2024-09-12T06:58:41.858 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -c image_ 2024-09-12T06:58:41.892 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 0 = 0 2024-09-12T06:58:41.892 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ error_code=0 2024-09-12T06:58:41.892 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set -e 2024-09-12T06:58:41.893 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 -eq 0 ']' 2024-09-12T06:58:41.893 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:58:41.893 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirror_leader image_map 2024-09-12T06:58:41.893 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:58:41.893 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:58:41.893 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local obj_name=rbd_mirror_leader 2024-09-12T06:58:41.893 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local filter=image_map 2024-09-12T06:58:41.893 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-12T06:58:41.893 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:58:41.893 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set +e 2024-09-12T06:58:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirror_leader image_map 2024-09-12T06:58:41.895 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:58:41.895 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:58:41.895 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local obj_name=rbd_mirror_leader 2024-09-12T06:58:41.895 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local filter=image_map 2024-09-12T06:58:41.895 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ list_omap_keys cluster1 mirror rbd_mirror_leader 2024-09-12T06:58:41.895 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:58:41.895 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:58:41.895 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local obj_name=rbd_mirror_leader 2024-09-12T06:58:41.895 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirror_leader 2024-09-12T06:58:41.895 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -c image_map 2024-09-12T06:58:41.932 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 0 = 0 2024-09-12T06:58:41.932 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ error_code=0 2024-09-12T06:58:41.933 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set -e 2024-09-12T06:58:41.933 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 -eq 0 ']' 2024-09-12T06:58:41.933 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:58:41.933 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_image_in_omap cluster2 mirror 2024-09-12T06:58:41.933 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:58:41.933 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:58:41.934 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirroring status_global 2024-09-12T06:58:41.934 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:58:41.934 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:58:41.934 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local obj_name=rbd_mirroring 2024-09-12T06:58:41.934 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local filter=status_global 2024-09-12T06:58:41.934 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-12T06:58:41.935 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:58:41.935 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set +e 2024-09-12T06:58:41.935 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirroring status_global 2024-09-12T06:58:41.935 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:58:41.935 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:58:41.935 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local obj_name=rbd_mirroring 2024-09-12T06:58:41.935 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local filter=status_global 2024-09-12T06:58:41.936 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ list_omap_keys cluster2 mirror rbd_mirroring 2024-09-12T06:58:41.936 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:58:41.936 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:58:41.936 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local obj_name=rbd_mirroring 2024-09-12T06:58:41.936 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirroring 2024-09-12T06:58:41.936 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -c status_global 2024-09-12T06:58:41.967 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 0 = 0 2024-09-12T06:58:41.967 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ error_code=0 2024-09-12T06:58:41.967 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set -e 2024-09-12T06:58:41.967 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 -eq 0 ']' 2024-09-12T06:58:41.967 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:58:41.967 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirroring image_ 2024-09-12T06:58:41.967 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:58:41.967 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:58:41.967 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local obj_name=rbd_mirroring 2024-09-12T06:58:41.968 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local filter=image_ 2024-09-12T06:58:41.968 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-12T06:58:41.968 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:58:41.968 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set +e 2024-09-12T06:58:41.968 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirroring image_ 2024-09-12T06:58:41.968 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:58:41.968 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:58:41.968 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local obj_name=rbd_mirroring 2024-09-12T06:58:41.968 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local filter=image_ 2024-09-12T06:58:41.968 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ list_omap_keys cluster2 mirror rbd_mirroring 2024-09-12T06:58:41.968 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:58:41.968 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:58:41.968 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local obj_name=rbd_mirroring 2024-09-12T06:58:41.969 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirroring 2024-09-12T06:58:41.969 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -c image_ 2024-09-12T06:58:41.999 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 0 = 0 2024-09-12T06:58:42.000 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ error_code=0 2024-09-12T06:58:42.000 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set -e 2024-09-12T06:58:42.000 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 -eq 0 ']' 2024-09-12T06:58:42.000 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:58:42.000 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirror_leader image_map 2024-09-12T06:58:42.000 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:58:42.000 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pool=mirror 2024-09-12T06:58:42.000 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local obj_name=rbd_mirror_leader 2024-09-12T06:58:42.000 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local filter=image_map 2024-09-12T06:58:42.000 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-12T06:58:42.000 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 0 2024-09-12T06:58:42.000 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set +e 2024-09-12T06:58:42.001 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirror_leader image_map 2024-09-12T06:58:42.002 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:58:42.002 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:58:42.002 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local obj_name=rbd_mirror_leader 2024-09-12T06:58:42.002 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local filter=image_map 2024-09-12T06:58:42.002 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ list_omap_keys cluster2 mirror rbd_mirror_leader 2024-09-12T06:58:42.002 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:58:42.002 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ grep -c image_map 2024-09-12T06:58:42.002 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local pool=mirror 2024-09-12T06:58:42.002 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local obj_name=rbd_mirror_leader 2024-09-12T06:58:42.002 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirror_leader 2024-09-12T06:58:42.034 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test 0 = 0 2024-09-12T06:58:42.035 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ error_code=0 2024-09-12T06:58:42.035 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set -e 2024-09-12T06:58:42.035 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 -eq 0 ']' 2024-09-12T06:58:42.035 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ return 0 2024-09-12T06:58:42.035 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ cleanup 0 2024-09-12T06:58:42.035 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local error_code=0 2024-09-12T06:58:42.035 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ set +e 2024-09-12T06:58:42.035 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 -ne 0 ']' 2024-09-12T06:58:42.035 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -z '' ']' 2024-09-12T06:58:42.035 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster instance 2024-09-12T06:58:42.035 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ CEPH_ARGS= 2024-09-12T06:58:42.035 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --cluster cluster1 osd pool rm mirror mirror --yes-i-really-really-mean-it 2024-09-12T06:58:43.060 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:pool 'mirror' does not exist 2024-09-12T06:58:43.071 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ CEPH_ARGS= 2024-09-12T06:58:43.071 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --cluster cluster2 osd pool rm mirror mirror --yes-i-really-really-mean-it 2024-09-12T06:58:44.283 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:pool 'mirror' does not exist 2024-09-12T06:58:44.299 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ CEPH_ARGS= 2024-09-12T06:58:44.299 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --cluster cluster1 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2024-09-12T06:58:45.079 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:pool 'mirror_parent' does not exist 2024-09-12T06:58:45.090 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ CEPH_ARGS= 2024-09-12T06:58:45.090 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ceph --cluster cluster2 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2024-09-12T06:58:46.221 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:pool 'mirror_parent' does not exist 2024-09-12T06:58:46.233 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2024-09-12T06:58:46.233 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ stop_mirrors cluster1 2024-09-12T06:58:46.233 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1 2024-09-12T06:58:46.233 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local sig= 2024-09-12T06:58:46.233 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:58:46.235 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:58:46.235 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ stop_mirror cluster1:0 '' 2024-09-12T06:58:46.236 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:0 2024-09-12T06:58:46.236 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local sig= 2024-09-12T06:58:46.236 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:58:46.236 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pid 2024-09-12T06:58:46.236 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_pid_file cluster1:0 2024-09-12T06:58:46.236 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1:0 2024-09-12T06:58:46.236 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:58:46.236 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:58:46.236 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:0 2024-09-12T06:58:46.236 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=cluster 2024-09-12T06:58:46.236 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:58:46.236 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:58:46.236 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:58:46.237 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T06:58:46.237 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval cluster=cluster1 2024-09-12T06:58:46.237 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ cluster=cluster1 2024-09-12T06:58:46.237 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=0 2024-09-12T06:58:46.237 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=0 2024-09-12T06:58:46.237 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'pid file' 2024-09-12T06:58:46.255 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2024-09-12T06:58:46.255 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2024-09-12T06:58:46.256 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ pid=37897 2024-09-12T06:58:46.256 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n 37897 ']' 2024-09-12T06:58:46.256 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ kill 37897 2024-09-12T06:58:46.256 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 16 32 2024-09-12T06:58:46.256 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:58:47.258 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ps auxww 2024-09-12T06:58:47.258 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ awk -v pid=37897 '$2 == pid {print; exit 1}' 2024-09-12T06:58:47.356 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ break 2024-09-12T06:58:47.357 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ps auxww 2024-09-12T06:58:47.357 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ awk -v pid=37897 '$2 == pid {print; exit 1}' 2024-09-12T06:58:47.366 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-12T06:58:47.366 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:58:47.366 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:58:47.366 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:58:47.366 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:58:47.366 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:58:47.366 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:58:47.366 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:58:47.366 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:58:47.366 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:58:47.366 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:58:47.367 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:58:47.367 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:58:47.367 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:58:47.367 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:58:47.367 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:58:47.387 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:58:47.387 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:58:47.390 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:0 cluster2 2024-09-12T06:58:47.390 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:0 2024-09-12T06:58:47.390 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:58:47.390 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:58:47.390 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T06:58:47.390 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:58:47.390 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:58:47.390 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:58:47.391 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:58:47.391 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:58:47.391 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:58:47.391 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:58:47.391 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:58:47.391 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:58:47.391 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:58:47.391 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T06:58:47.420 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:58:47.420 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T06:58:47.423 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_pid_file cluster1:0 2024-09-12T06:58:47.423 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:0 2024-09-12T06:58:47.423 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:58:47.423 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T06:58:47.423 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:0 2024-09-12T06:58:47.423 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:58:47.423 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:58:47.423 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:58:47.423 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:58:47.423 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T06:58:47.423 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:58:47.423 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:58:47.423 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:58:47.424 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:58:47.424 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'pid file' 2024-09-12T06:58:47.444 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2024-09-12T06:58:47.444 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2024-09-12T06:58:47.447 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:58:47.447 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ stop_mirror cluster1:1 '' 2024-09-12T06:58:47.447 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster1:1 2024-09-12T06:58:47.447 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local sig= 2024-09-12T06:58:47.447 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:58:47.447 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pid 2024-09-12T06:58:47.447 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_pid_file cluster1:1 2024-09-12T06:58:47.447 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster1:1 2024-09-12T06:58:47.447 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:58:47.447 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T06:58:47.447 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster1:1 2024-09-12T06:58:47.448 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=cluster 2024-09-12T06:58:47.448 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:58:47.448 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:58:47.448 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:58:47.448 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T06:58:47.448 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval cluster=cluster1 2024-09-12T06:58:47.448 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ cluster=cluster1 2024-09-12T06:58:47.448 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=1 2024-09-12T06:58:47.448 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=1 2024-09-12T06:58:47.448 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'pid file' 2024-09-12T06:58:47.466 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2024-09-12T06:58:47.466 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2024-09-12T06:58:47.468 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ pid=37941 2024-09-12T06:58:47.468 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n 37941 ']' 2024-09-12T06:58:47.468 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ kill 37941 2024-09-12T06:58:47.468 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 16 32 2024-09-12T06:58:47.468 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:58:48.469 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ps auxww 2024-09-12T06:58:48.470 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ awk -v pid=37941 '$2 == pid {print; exit 1}' 2024-09-12T06:58:48.481 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ break 2024-09-12T06:58:48.481 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ps auxww 2024-09-12T06:58:48.481 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ awk -v pid=37941 '$2 == pid {print; exit 1}' 2024-09-12T06:58:48.493 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-12T06:58:48.493 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:1 2024-09-12T06:58:48.493 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:58:48.493 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:58:48.493 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T06:58:48.494 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:1 2024-09-12T06:58:48.494 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:58:48.494 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:58:48.494 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:58:48.494 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:58:48.494 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T06:58:48.494 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:58:48.494 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:58:48.494 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=1 2024-09-12T06:58:48.494 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:58:48.494 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T06:58:48.514 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:58:48.514 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:58:48.516 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster1:1 cluster2 2024-09-12T06:58:48.516 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster1:1 2024-09-12T06:58:48.516 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:58:48.516 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:58:48.516 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T06:58:48.516 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:1 2024-09-12T06:58:48.516 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:58:48.517 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:58:48.517 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:58:48.517 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:58:48.517 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T06:58:48.517 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster1 2024-09-12T06:58:48.517 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster1 2024-09-12T06:58:48.517 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=1 2024-09-12T06:58:48.517 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:58:48.517 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T06:58:48.536 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:58:48.537 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T06:58:48.539 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_pid_file cluster1:1 2024-09-12T06:58:48.539 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1:1 2024-09-12T06:58:48.539 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:58:48.539 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T06:58:48.539 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster1:1 2024-09-12T06:58:48.539 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:58:48.539 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:58:48.539 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster1 2024-09-12T06:58:48.539 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:58:48.539 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T06:58:48.539 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster1 2024-09-12T06:58:48.539 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster1 2024-09-12T06:58:48.539 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=1 2024-09-12T06:58:48.539 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:58:48.540 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'pid file' 2024-09-12T06:58:48.559 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2024-09-12T06:58:48.560 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2024-09-12T06:58:48.561 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2024-09-12T06:58:48.561 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ stop_mirrors cluster2 2024-09-12T06:58:48.561 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2 2024-09-12T06:58:48.561 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local sig= 2024-09-12T06:58:48.561 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ seq 0 1 2024-09-12T06:58:48.563 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:58:48.563 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ stop_mirror cluster2:0 '' 2024-09-12T06:58:48.563 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2:0 2024-09-12T06:58:48.563 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local sig= 2024-09-12T06:58:48.563 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:58:48.564 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pid 2024-09-12T06:58:48.564 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_pid_file cluster2:0 2024-09-12T06:58:48.564 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster2:0 2024-09-12T06:58:48.564 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:58:48.564 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster2:0 cluster instance 2024-09-12T06:58:48.564 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster2:0 2024-09-12T06:58:48.564 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=cluster 2024-09-12T06:58:48.564 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:58:48.564 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster2 2024-09-12T06:58:48.564 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:58:48.564 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-12T06:58:48.564 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval cluster=cluster2 2024-09-12T06:58:48.564 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ cluster=cluster2 2024-09-12T06:58:48.565 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=0 2024-09-12T06:58:48.565 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=0 2024-09-12T06:58:48.565 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'pid file' 2024-09-12T06:58:48.583 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2024-09-12T06:58:48.583 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cat /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2024-09-12T06:58:48.585 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ pid=38002 2024-09-12T06:58:48.585 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n 38002 ']' 2024-09-12T06:58:48.585 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ kill 38002 2024-09-12T06:58:48.585 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 16 32 2024-09-12T06:58:48.585 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:58:49.586 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ps auxww 2024-09-12T06:58:49.618 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ awk -v pid=38002 '$2 == pid {print; exit 1}' 2024-09-12T06:58:49.621 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ break 2024-09-12T06:58:49.621 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ps auxww 2024-09-12T06:58:49.621 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ awk -v pid=38002 '$2 == pid {print; exit 1}' 2024-09-12T06:58:49.621 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster2:0 cluster1 2024-09-12T06:58:49.621 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster2:0 2024-09-12T06:58:49.621 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:58:49.621 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:58:49.621 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-12T06:58:49.621 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster2:0 2024-09-12T06:58:49.621 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:58:49.622 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:58:49.622 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster2 2024-09-12T06:58:49.622 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:58:49.622 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-12T06:58:49.622 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster2 2024-09-12T06:58:49.622 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster2 2024-09-12T06:58:49.622 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:58:49.622 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:58:49.622 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-12T06:58:49.632 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-12T06:58:49.632 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-12T06:58:49.634 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-09-12T06:58:49.635 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster2:0 2024-09-12T06:58:49.635 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:58:49.635 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:58:49.635 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-12T06:58:49.635 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster2:0 2024-09-12T06:58:49.635 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:58:49.635 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:58:49.635 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster2 2024-09-12T06:58:49.635 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:58:49.635 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-12T06:58:49.635 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster2 2024-09-12T06:58:49.636 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster2 2024-09-12T06:58:49.636 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:58:49.636 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:58:49.636 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-12T06:58:49.655 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-12T06:58:49.655 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-12T06:58:49.658 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_pid_file cluster2:0 2024-09-12T06:58:49.658 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2:0 2024-09-12T06:58:49.658 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:58:49.658 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-12T06:58:49.658 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster2:0 2024-09-12T06:58:49.658 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:58:49.658 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:58:49.658 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster2 2024-09-12T06:58:49.658 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=0 2024-09-12T06:58:49.659 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-12T06:58:49.659 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster2 2024-09-12T06:58:49.659 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster2 2024-09-12T06:58:49.659 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=0 2024-09-12T06:58:49.659 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=0 2024-09-12T06:58:49.659 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'pid file' 2024-09-12T06:58:49.678 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2024-09-12T06:58:49.678 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2024-09-12T06:58:49.680 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T06:58:49.680 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ stop_mirror cluster2:1 '' 2024-09-12T06:58:49.680 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local cluster=cluster2:1 2024-09-12T06:58:49.680 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local sig= 2024-09-12T06:58:49.681 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test -n '' 2024-09-12T06:58:49.681 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ local pid 2024-09-12T06:58:49.681 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ daemon_pid_file cluster2:1 2024-09-12T06:58:49.681 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster=cluster2:1 2024-09-12T06:58:49.681 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance 2024-09-12T06:58:49.681 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ set_cluster_instance cluster2:1 cluster instance 2024-09-12T06:58:49.681 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local val=cluster2:1 2024-09-12T06:58:49.681 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local cluster_var_name=cluster 2024-09-12T06:58:49.681 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local instance_var_name=instance 2024-09-12T06:58:49.681 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster2 2024-09-12T06:58:49.681 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:58:49.682 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ '[' 1 = cluster2:1 ']' 2024-09-12T06:58:49.682 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval cluster=cluster2 2024-09-12T06:58:49.682 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ cluster=cluster2 2024-09-12T06:58:49.682 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ eval instance=1 2024-09-12T06:58:49.682 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ instance=1 2024-09-12T06:58:49.682 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.1 'pid file' 2024-09-12T06:58:49.700 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid 2024-09-12T06:58:49.700 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cat /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid 2024-09-12T06:58:49.701 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ pid=38047 2024-09-12T06:58:49.701 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -n 38047 ']' 2024-09-12T06:58:49.701 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ kill 38047 2024-09-12T06:58:49.701 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ for s in 1 2 4 8 16 32 2024-09-12T06:58:49.702 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ sleep 1 2024-09-12T06:58:50.703 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ps auxww 2024-09-12T06:58:50.703 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ awk -v pid=38047 '$2 == pid {print; exit 1}' 2024-09-12T06:58:50.714 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ break 2024-09-12T06:58:50.714 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ ps auxww 2024-09-12T06:58:50.714 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ awk -v pid=38047 '$2 == pid {print; exit 1}' 2024-09-12T06:58:50.726 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster2:1 cluster1 2024-09-12T06:58:50.726 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster2:1 2024-09-12T06:58:50.726 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster1 2024-09-12T06:58:50.726 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:58:50.726 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2024-09-12T06:58:50.726 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster2:1 2024-09-12T06:58:50.726 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:58:50.727 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:58:50.727 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster2 2024-09-12T06:58:50.727 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:58:50.727 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 1 = cluster2:1 ']' 2024-09-12T06:58:50.727 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster2 2024-09-12T06:58:50.727 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster2 2024-09-12T06:58:50.727 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=1 2024-09-12T06:58:50.727 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:58:50.727 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2024-09-12T06:58:50.747 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-12T06:58:50.747 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-12T06:58:50.750 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_asok_file cluster2:1 cluster2 2024-09-12T06:58:50.750 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local local_cluster=cluster2:1 2024-09-12T06:58:50.751 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2 2024-09-12T06:58:50.751 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:58:50.751 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2024-09-12T06:58:50.751 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster2:1 2024-09-12T06:58:50.751 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=local_cluster 2024-09-12T06:58:50.751 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:58:50.751 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster2 2024-09-12T06:58:50.751 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:58:50.751 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 1 = cluster2:1 ']' 2024-09-12T06:58:50.751 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval local_cluster=cluster2 2024-09-12T06:58:50.751 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ local_cluster=cluster2 2024-09-12T06:58:50.751 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=1 2024-09-12T06:58:50.751 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:58:50.752 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2024-09-12T06:58:50.770 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-12T06:58:50.771 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-12T06:58:50.773 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ daemon_pid_file cluster2:1 2024-09-12T06:58:50.773 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster=cluster2:1 2024-09-12T06:58:50.773 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance 2024-09-12T06:58:50.773 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ set_cluster_instance cluster2:1 cluster instance 2024-09-12T06:58:50.773 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local val=cluster2:1 2024-09-12T06:58:50.773 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local cluster_var_name=cluster 2024-09-12T06:58:50.774 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ local instance_var_name=instance 2024-09-12T06:58:50.774 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ cluster=cluster2 2024-09-12T06:58:50.774 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ instance=1 2024-09-12T06:58:50.774 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ '[' 1 = cluster2:1 ']' 2024-09-12T06:58:50.774 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval cluster=cluster2 2024-09-12T06:58:50.774 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ cluster=cluster2 2024-09-12T06:58:50.774 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ eval instance=1 2024-09-12T06:58:50.774 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ instance=1 2024-09-12T06:58:50.774 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'pid file' 2024-09-12T06:58:50.793 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid 2024-09-12T06:58:50.793 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid 2024-09-12T06:58:50.794 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' -z 1 ']' 2024-09-12T06:58:50.794 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ test '' = /tmp/tmp.GzjHZGs9Z0 2024-09-12T06:58:50.794 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ rm -Rf /tmp/tmp.GzjHZGs9Z0 2024-09-12T06:58:50.803 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ '[' 0 -eq 0 ']' 2024-09-12T06:58:50.803 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ echo OK 2024-09-12T06:58:50.803 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stdout:OK 2024-09-12T06:58:50.805 INFO:tasks.workunit.cluster1.client.mirror.smithi152.stderr:+ exit 0 2024-09-12T06:58:50.805 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-12T06:58:50.805 DEBUG:teuthology.orchestra.run.smithi152:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp 2024-09-12T06:58:50.979 INFO:tasks.workunit:Stopping ['rbd/rbd_mirror_stress.sh'] on cluster1.client.mirror... 2024-09-12T06:58:50.979 DEBUG:teuthology.orchestra.run.smithi152:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.cluster1.client.mirror /home/ubuntu/cephtest/clone.cluster1.client.mirror 2024-09-12T06:58:51.226 DEBUG:teuthology.parallel:result is None 2024-09-12T06:58:51.227 DEBUG:teuthology.orchestra.run.smithi152:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-09-12T06:58:51.252 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-09-12T06:58:51.252 DEBUG:teuthology.orchestra.run.smithi152:> rmdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-09-12T06:58:51.315 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-09-12T06:58:51.315 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-12T06:58:51.327 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2024-09-12T06:58:51.327 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T06:58:51.636 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:58:51.637 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-12T06:58:51.649 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":1332,"stamp":"2024-09-12T06:58:50.679973+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":146,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2562864,"kb_used_data":14324,"kb_used_omap":393,"kb_used_meta":2548086,"kb_avail":280552656,"statfs":{"total":289910292480,"available":287285919744,"internally_reserved":0,"allocated":14667776,"data_stored":8064477,"data_compressed":11786,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":402601,"internal_metadata":2609240919},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-426701878,"num_objects":-143,"num_object_clones":-1,"num_object_copies":-286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-143,"num_whiteouts":-1,"num_read":-802928,"num_read_kb":-43785596,"num_write":-1171179,"num_write_kb":-23509624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-6,"num_bytes_recovered":-27733885,"num_keys_recovered":0,"num_objects_omap":-15,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001808"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":824,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-12T06:58:44.694032+0000","last_change":"2024-09-12T06:28:13.359110+0000","last_active":"2024-09-12T06:58:44.694032+0000","last_peered":"2024-09-12T06:58:44.694032+0000","last_clean":"2024-09-12T06:58:44.694032+0000","last_became_active":"2024-09-12T06:28:11.358431+0000","last_became_peered":"2024-09-12T06:28:11.358431+0000","last_unstale":"2024-09-12T06:58:44.694032+0000","last_undegraded":"2024-09-12T06:58:44.694032+0000","last_fullsized":"2024-09-12T06:58:44.694032+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_clean_scrub_stamp":"2024-09-12T06:28:10.346475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:44.092579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089666099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":824,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-12T06:58:44.698017+0000","last_change":"2024-09-12T06:28:13.359336+0000","last_active":"2024-09-12T06:58:44.698017+0000","last_peered":"2024-09-12T06:58:44.698017+0000","last_clean":"2024-09-12T06:58:44.698017+0000","last_became_active":"2024-09-12T06:28:11.358327+0000","last_became_peered":"2024-09-12T06:28:11.358327+0000","last_unstale":"2024-09-12T06:58:44.698017+0000","last_undegraded":"2024-09-12T06:58:44.698017+0000","last_fullsized":"2024-09-12T06:58:44.698017+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_clean_scrub_stamp":"2024-09-12T06:28:10.346475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:08:25.715384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001063589,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":824,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-12T06:58:44.701263+0000","last_change":"2024-09-12T06:28:13.358717+0000","last_active":"2024-09-12T06:58:44.701263+0000","last_peered":"2024-09-12T06:58:44.701263+0000","last_clean":"2024-09-12T06:58:44.701263+0000","last_became_active":"2024-09-12T06:28:11.358302+0000","last_became_peered":"2024-09-12T06:28:11.358302+0000","last_unstale":"2024-09-12T06:58:44.701263+0000","last_undegraded":"2024-09-12T06:58:44.701263+0000","last_fullsized":"2024-09-12T06:58:44.701263+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_clean_scrub_stamp":"2024-09-12T06:28:10.346475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T17:13:31.101064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055283899999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":824,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-12T06:58:44.693709+0000","last_change":"2024-09-12T06:28:13.358819+0000","last_active":"2024-09-12T06:58:44.693709+0000","last_peered":"2024-09-12T06:58:44.693709+0000","last_clean":"2024-09-12T06:58:44.693709+0000","last_became_active":"2024-09-12T06:28:11.357578+0000","last_became_peered":"2024-09-12T06:28:11.357578+0000","last_unstale":"2024-09-12T06:58:44.693709+0000","last_undegraded":"2024-09-12T06:58:44.693709+0000","last_fullsized":"2024-09-12T06:58:44.693709+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_clean_scrub_stamp":"2024-09-12T06:28:10.346475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:54:00.421871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000609568,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":828,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-12T06:58:46.112984+0000","last_change":"2024-09-12T06:28:13.364918+0000","last_active":"2024-09-12T06:58:46.112984+0000","last_peered":"2024-09-12T06:58:46.112984+0000","last_clean":"2024-09-12T06:58:46.112984+0000","last_became_active":"2024-09-12T06:28:11.357233+0000","last_became_peered":"2024-09-12T06:28:11.357233+0000","last_unstale":"2024-09-12T06:58:46.112984+0000","last_undegraded":"2024-09-12T06:58:46.112984+0000","last_fullsized":"2024-09-12T06:58:46.112984+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_clean_scrub_stamp":"2024-09-12T06:28:10.346475+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:37:00.884436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012297149999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":826,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-12T06:58:47.878327+0000","last_change":"2024-09-12T06:28:13.473086+0000","last_active":"2024-09-12T06:58:47.878327+0000","last_peered":"2024-09-12T06:58:47.878327+0000","last_clean":"2024-09-12T06:58:47.878327+0000","last_became_active":"2024-09-12T06:28:11.357446+0000","last_became_peered":"2024-09-12T06:28:11.357446+0000","last_unstale":"2024-09-12T06:58:47.878327+0000","last_undegraded":"2024-09-12T06:58:47.878327+0000","last_fullsized":"2024-09-12T06:58:47.878327+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_clean_scrub_stamp":"2024-09-12T06:28:10.346475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T16:51:04.766222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041499400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":827,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-12T06:58:47.878869+0000","last_change":"2024-09-12T06:28:13.473086+0000","last_active":"2024-09-12T06:58:47.878869+0000","last_peered":"2024-09-12T06:58:47.878869+0000","last_clean":"2024-09-12T06:58:47.878869+0000","last_became_active":"2024-09-12T06:28:11.359644+0000","last_became_peered":"2024-09-12T06:28:11.359644+0000","last_unstale":"2024-09-12T06:58:47.878869+0000","last_undegraded":"2024-09-12T06:58:47.878869+0000","last_fullsized":"2024-09-12T06:58:47.878869+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_clean_scrub_stamp":"2024-09-12T06:28:10.346475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T16:48:40.580401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000404719,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":825,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-12T06:58:44.697090+0000","last_change":"2024-09-12T06:28:13.359050+0000","last_active":"2024-09-12T06:58:44.697090+0000","last_peered":"2024-09-12T06:58:44.697090+0000","last_clean":"2024-09-12T06:58:44.697090+0000","last_became_active":"2024-09-12T06:28:11.358377+0000","last_became_peered":"2024-09-12T06:28:11.358377+0000","last_unstale":"2024-09-12T06:58:44.697090+0000","last_undegraded":"2024-09-12T06:58:44.697090+0000","last_fullsized":"2024-09-12T06:58:44.697090+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_clean_scrub_stamp":"2024-09-12T06:28:10.346475+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:05:39.794241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084816700000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'52","reported_seq":893,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-12T06:58:44.694153+0000","last_change":"2024-09-12T06:28:06.474810+0000","last_active":"2024-09-12T06:58:44.694153+0000","last_peered":"2024-09-12T06:58:44.694153+0000","last_clean":"2024-09-12T06:58:44.694153+0000","last_became_active":"2024-09-12T06:28:06.473701+0000","last_became_peered":"2024-09-12T06:28:06.473701+0000","last_unstale":"2024-09-12T06:58:44.694153+0000","last_undegraded":"2024-09-12T06:58:44.694153+0000","last_fullsized":"2024-09-12T06:58:44.694153+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:28:05.332439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:28:05.332439+0000","last_clean_scrub_stamp":"2024-09-12T06:28:05.332439+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T16:09:15.324127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11786,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738743,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703004,"kb_used_data":4744,"kb_used_omap":137,"kb_used_meta":698102,"kb_avail":93668836,"statfs":{"total":96636764160,"available":95916888064,"internally_reserved":0,"allocated":4857856,"data_stored":2298988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":140613,"internal_metadata":714857147},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738741,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":833548,"kb_used_data":4792,"kb_used_omap":115,"kb_used_meta":828620,"kb_avail":93538292,"statfs":{"total":96636764160,"available":95783211008,"internally_reserved":0,"allocated":4907008,"data_stored":2882754,"data_compressed":5893,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":118385,"internal_metadata":848507279},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738742,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1026312,"kb_used_data":4788,"kb_used_omap":140,"kb_used_meta":1021363,"kb_avail":93345528,"statfs":{"total":96636764160,"available":95585820672,"internally_reserved":0,"allocated":4902912,"data_stored":2882735,"data_compressed":5893,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":143603,"internal_metadata":1045876493},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5893,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5893,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T06:58:51.650 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T06:58:51.931 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:58:51.932 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-12T06:58:51.944 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":1332,"stamp":"2024-09-12T06:58:50.679973+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":146,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2562864,"kb_used_data":14324,"kb_used_omap":393,"kb_used_meta":2548086,"kb_avail":280552656,"statfs":{"total":289910292480,"available":287285919744,"internally_reserved":0,"allocated":14667776,"data_stored":8064477,"data_compressed":11786,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":402601,"internal_metadata":2609240919},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-426701878,"num_objects":-143,"num_object_clones":-1,"num_object_copies":-286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-143,"num_whiteouts":-1,"num_read":-802928,"num_read_kb":-43785596,"num_write":-1171179,"num_write_kb":-23509624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-6,"num_bytes_recovered":-27733885,"num_keys_recovered":0,"num_objects_omap":-15,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001808"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":824,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-12T06:58:44.694032+0000","last_change":"2024-09-12T06:28:13.359110+0000","last_active":"2024-09-12T06:58:44.694032+0000","last_peered":"2024-09-12T06:58:44.694032+0000","last_clean":"2024-09-12T06:58:44.694032+0000","last_became_active":"2024-09-12T06:28:11.358431+0000","last_became_peered":"2024-09-12T06:28:11.358431+0000","last_unstale":"2024-09-12T06:58:44.694032+0000","last_undegraded":"2024-09-12T06:58:44.694032+0000","last_fullsized":"2024-09-12T06:58:44.694032+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_clean_scrub_stamp":"2024-09-12T06:28:10.346475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:44.092579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089666099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":824,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-12T06:58:44.698017+0000","last_change":"2024-09-12T06:28:13.359336+0000","last_active":"2024-09-12T06:58:44.698017+0000","last_peered":"2024-09-12T06:58:44.698017+0000","last_clean":"2024-09-12T06:58:44.698017+0000","last_became_active":"2024-09-12T06:28:11.358327+0000","last_became_peered":"2024-09-12T06:28:11.358327+0000","last_unstale":"2024-09-12T06:58:44.698017+0000","last_undegraded":"2024-09-12T06:58:44.698017+0000","last_fullsized":"2024-09-12T06:58:44.698017+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_clean_scrub_stamp":"2024-09-12T06:28:10.346475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:08:25.715384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001063589,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":824,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-12T06:58:44.701263+0000","last_change":"2024-09-12T06:28:13.358717+0000","last_active":"2024-09-12T06:58:44.701263+0000","last_peered":"2024-09-12T06:58:44.701263+0000","last_clean":"2024-09-12T06:58:44.701263+0000","last_became_active":"2024-09-12T06:28:11.358302+0000","last_became_peered":"2024-09-12T06:28:11.358302+0000","last_unstale":"2024-09-12T06:58:44.701263+0000","last_undegraded":"2024-09-12T06:58:44.701263+0000","last_fullsized":"2024-09-12T06:58:44.701263+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_clean_scrub_stamp":"2024-09-12T06:28:10.346475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T17:13:31.101064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055283899999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":824,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-12T06:58:44.693709+0000","last_change":"2024-09-12T06:28:13.358819+0000","last_active":"2024-09-12T06:58:44.693709+0000","last_peered":"2024-09-12T06:58:44.693709+0000","last_clean":"2024-09-12T06:58:44.693709+0000","last_became_active":"2024-09-12T06:28:11.357578+0000","last_became_peered":"2024-09-12T06:28:11.357578+0000","last_unstale":"2024-09-12T06:58:44.693709+0000","last_undegraded":"2024-09-12T06:58:44.693709+0000","last_fullsized":"2024-09-12T06:58:44.693709+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_clean_scrub_stamp":"2024-09-12T06:28:10.346475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:54:00.421871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000609568,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":828,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-12T06:58:46.112984+0000","last_change":"2024-09-12T06:28:13.364918+0000","last_active":"2024-09-12T06:58:46.112984+0000","last_peered":"2024-09-12T06:58:46.112984+0000","last_clean":"2024-09-12T06:58:46.112984+0000","last_became_active":"2024-09-12T06:28:11.357233+0000","last_became_peered":"2024-09-12T06:28:11.357233+0000","last_unstale":"2024-09-12T06:58:46.112984+0000","last_undegraded":"2024-09-12T06:58:46.112984+0000","last_fullsized":"2024-09-12T06:58:46.112984+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_clean_scrub_stamp":"2024-09-12T06:28:10.346475+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:37:00.884436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012297149999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":826,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-12T06:58:47.878327+0000","last_change":"2024-09-12T06:28:13.473086+0000","last_active":"2024-09-12T06:58:47.878327+0000","last_peered":"2024-09-12T06:58:47.878327+0000","last_clean":"2024-09-12T06:58:47.878327+0000","last_became_active":"2024-09-12T06:28:11.357446+0000","last_became_peered":"2024-09-12T06:28:11.357446+0000","last_unstale":"2024-09-12T06:58:47.878327+0000","last_undegraded":"2024-09-12T06:58:47.878327+0000","last_fullsized":"2024-09-12T06:58:47.878327+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_clean_scrub_stamp":"2024-09-12T06:28:10.346475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T16:51:04.766222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041499400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":827,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-12T06:58:47.878869+0000","last_change":"2024-09-12T06:28:13.473086+0000","last_active":"2024-09-12T06:58:47.878869+0000","last_peered":"2024-09-12T06:58:47.878869+0000","last_clean":"2024-09-12T06:58:47.878869+0000","last_became_active":"2024-09-12T06:28:11.359644+0000","last_became_peered":"2024-09-12T06:28:11.359644+0000","last_unstale":"2024-09-12T06:58:47.878869+0000","last_undegraded":"2024-09-12T06:58:47.878869+0000","last_fullsized":"2024-09-12T06:58:47.878869+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_clean_scrub_stamp":"2024-09-12T06:28:10.346475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T16:48:40.580401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000404719,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":825,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-12T06:58:44.697090+0000","last_change":"2024-09-12T06:28:13.359050+0000","last_active":"2024-09-12T06:58:44.697090+0000","last_peered":"2024-09-12T06:58:44.697090+0000","last_clean":"2024-09-12T06:58:44.697090+0000","last_became_active":"2024-09-12T06:28:11.358377+0000","last_became_peered":"2024-09-12T06:28:11.358377+0000","last_unstale":"2024-09-12T06:58:44.697090+0000","last_undegraded":"2024-09-12T06:58:44.697090+0000","last_fullsized":"2024-09-12T06:58:44.697090+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_clean_scrub_stamp":"2024-09-12T06:28:10.346475+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:05:39.794241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084816700000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'52","reported_seq":893,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-12T06:58:44.694153+0000","last_change":"2024-09-12T06:28:06.474810+0000","last_active":"2024-09-12T06:58:44.694153+0000","last_peered":"2024-09-12T06:58:44.694153+0000","last_clean":"2024-09-12T06:58:44.694153+0000","last_became_active":"2024-09-12T06:28:06.473701+0000","last_became_peered":"2024-09-12T06:28:06.473701+0000","last_unstale":"2024-09-12T06:58:44.694153+0000","last_undegraded":"2024-09-12T06:58:44.694153+0000","last_fullsized":"2024-09-12T06:58:44.694153+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:28:05.332439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:28:05.332439+0000","last_clean_scrub_stamp":"2024-09-12T06:28:05.332439+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T16:09:15.324127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11786,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738743,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703004,"kb_used_data":4744,"kb_used_omap":137,"kb_used_meta":698102,"kb_avail":93668836,"statfs":{"total":96636764160,"available":95916888064,"internally_reserved":0,"allocated":4857856,"data_stored":2298988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":140613,"internal_metadata":714857147},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738741,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":833548,"kb_used_data":4792,"kb_used_omap":115,"kb_used_meta":828620,"kb_avail":93538292,"statfs":{"total":96636764160,"available":95783211008,"internally_reserved":0,"allocated":4907008,"data_stored":2882754,"data_compressed":5893,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":118385,"internal_metadata":848507279},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738742,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1026312,"kb_used_data":4788,"kb_used_omap":140,"kb_used_meta":1021363,"kb_avail":93345528,"statfs":{"total":96636764160,"available":95585820672,"internally_reserved":0,"allocated":4902912,"data_stored":2882735,"data_compressed":5893,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":143603,"internal_metadata":1045876493},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5893,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5893,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T06:58:51.945 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2024-09-12T06:58:51.945 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T06:58:52.229 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:58:52.229 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-12T06:58:52.241 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":1332,"stamp":"2024-09-12T06:58:50.679973+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":146,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2562864,"kb_used_data":14324,"kb_used_omap":393,"kb_used_meta":2548086,"kb_avail":280552656,"statfs":{"total":289910292480,"available":287285919744,"internally_reserved":0,"allocated":14667776,"data_stored":8064477,"data_compressed":11786,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":402601,"internal_metadata":2609240919},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-426701878,"num_objects":-143,"num_object_clones":-1,"num_object_copies":-286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-143,"num_whiteouts":-1,"num_read":-802928,"num_read_kb":-43785596,"num_write":-1171179,"num_write_kb":-23509624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-6,"num_bytes_recovered":-27733885,"num_keys_recovered":0,"num_objects_omap":-15,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001808"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":824,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-12T06:58:44.694032+0000","last_change":"2024-09-12T06:28:13.359110+0000","last_active":"2024-09-12T06:58:44.694032+0000","last_peered":"2024-09-12T06:58:44.694032+0000","last_clean":"2024-09-12T06:58:44.694032+0000","last_became_active":"2024-09-12T06:28:11.358431+0000","last_became_peered":"2024-09-12T06:28:11.358431+0000","last_unstale":"2024-09-12T06:58:44.694032+0000","last_undegraded":"2024-09-12T06:58:44.694032+0000","last_fullsized":"2024-09-12T06:58:44.694032+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_clean_scrub_stamp":"2024-09-12T06:28:10.346475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:44.092579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089666099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":824,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-12T06:58:44.698017+0000","last_change":"2024-09-12T06:28:13.359336+0000","last_active":"2024-09-12T06:58:44.698017+0000","last_peered":"2024-09-12T06:58:44.698017+0000","last_clean":"2024-09-12T06:58:44.698017+0000","last_became_active":"2024-09-12T06:28:11.358327+0000","last_became_peered":"2024-09-12T06:28:11.358327+0000","last_unstale":"2024-09-12T06:58:44.698017+0000","last_undegraded":"2024-09-12T06:58:44.698017+0000","last_fullsized":"2024-09-12T06:58:44.698017+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_clean_scrub_stamp":"2024-09-12T06:28:10.346475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:08:25.715384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001063589,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":824,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-12T06:58:44.701263+0000","last_change":"2024-09-12T06:28:13.358717+0000","last_active":"2024-09-12T06:58:44.701263+0000","last_peered":"2024-09-12T06:58:44.701263+0000","last_clean":"2024-09-12T06:58:44.701263+0000","last_became_active":"2024-09-12T06:28:11.358302+0000","last_became_peered":"2024-09-12T06:28:11.358302+0000","last_unstale":"2024-09-12T06:58:44.701263+0000","last_undegraded":"2024-09-12T06:58:44.701263+0000","last_fullsized":"2024-09-12T06:58:44.701263+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_clean_scrub_stamp":"2024-09-12T06:28:10.346475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T17:13:31.101064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055283899999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":824,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-12T06:58:44.693709+0000","last_change":"2024-09-12T06:28:13.358819+0000","last_active":"2024-09-12T06:58:44.693709+0000","last_peered":"2024-09-12T06:58:44.693709+0000","last_clean":"2024-09-12T06:58:44.693709+0000","last_became_active":"2024-09-12T06:28:11.357578+0000","last_became_peered":"2024-09-12T06:28:11.357578+0000","last_unstale":"2024-09-12T06:58:44.693709+0000","last_undegraded":"2024-09-12T06:58:44.693709+0000","last_fullsized":"2024-09-12T06:58:44.693709+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_clean_scrub_stamp":"2024-09-12T06:28:10.346475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:54:00.421871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000609568,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":828,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-12T06:58:46.112984+0000","last_change":"2024-09-12T06:28:13.364918+0000","last_active":"2024-09-12T06:58:46.112984+0000","last_peered":"2024-09-12T06:58:46.112984+0000","last_clean":"2024-09-12T06:58:46.112984+0000","last_became_active":"2024-09-12T06:28:11.357233+0000","last_became_peered":"2024-09-12T06:28:11.357233+0000","last_unstale":"2024-09-12T06:58:46.112984+0000","last_undegraded":"2024-09-12T06:58:46.112984+0000","last_fullsized":"2024-09-12T06:58:46.112984+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_clean_scrub_stamp":"2024-09-12T06:28:10.346475+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:37:00.884436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012297149999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":826,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-12T06:58:47.878327+0000","last_change":"2024-09-12T06:28:13.473086+0000","last_active":"2024-09-12T06:58:47.878327+0000","last_peered":"2024-09-12T06:58:47.878327+0000","last_clean":"2024-09-12T06:58:47.878327+0000","last_became_active":"2024-09-12T06:28:11.357446+0000","last_became_peered":"2024-09-12T06:28:11.357446+0000","last_unstale":"2024-09-12T06:58:47.878327+0000","last_undegraded":"2024-09-12T06:58:47.878327+0000","last_fullsized":"2024-09-12T06:58:47.878327+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_clean_scrub_stamp":"2024-09-12T06:28:10.346475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T16:51:04.766222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041499400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":827,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-12T06:58:47.878869+0000","last_change":"2024-09-12T06:28:13.473086+0000","last_active":"2024-09-12T06:58:47.878869+0000","last_peered":"2024-09-12T06:58:47.878869+0000","last_clean":"2024-09-12T06:58:47.878869+0000","last_became_active":"2024-09-12T06:28:11.359644+0000","last_became_peered":"2024-09-12T06:28:11.359644+0000","last_unstale":"2024-09-12T06:58:47.878869+0000","last_undegraded":"2024-09-12T06:58:47.878869+0000","last_fullsized":"2024-09-12T06:58:47.878869+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_clean_scrub_stamp":"2024-09-12T06:28:10.346475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T16:48:40.580401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000404719,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":825,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-12T06:58:44.697090+0000","last_change":"2024-09-12T06:28:13.359050+0000","last_active":"2024-09-12T06:58:44.697090+0000","last_peered":"2024-09-12T06:58:44.697090+0000","last_clean":"2024-09-12T06:58:44.697090+0000","last_became_active":"2024-09-12T06:28:11.358377+0000","last_became_peered":"2024-09-12T06:28:11.358377+0000","last_unstale":"2024-09-12T06:58:44.697090+0000","last_undegraded":"2024-09-12T06:58:44.697090+0000","last_fullsized":"2024-09-12T06:58:44.697090+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_clean_scrub_stamp":"2024-09-12T06:28:10.346475+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:05:39.794241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084816700000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'52","reported_seq":893,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-09-12T06:58:44.694153+0000","last_change":"2024-09-12T06:28:06.474810+0000","last_active":"2024-09-12T06:58:44.694153+0000","last_peered":"2024-09-12T06:58:44.694153+0000","last_clean":"2024-09-12T06:58:44.694153+0000","last_became_active":"2024-09-12T06:28:06.473701+0000","last_became_peered":"2024-09-12T06:28:06.473701+0000","last_unstale":"2024-09-12T06:58:44.694153+0000","last_undegraded":"2024-09-12T06:58:44.694153+0000","last_fullsized":"2024-09-12T06:58:44.694153+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:28:05.332439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:28:05.332439+0000","last_clean_scrub_stamp":"2024-09-12T06:28:05.332439+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T16:09:15.324127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11786,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738743,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703004,"kb_used_data":4744,"kb_used_omap":137,"kb_used_meta":698102,"kb_avail":93668836,"statfs":{"total":96636764160,"available":95916888064,"internally_reserved":0,"allocated":4857856,"data_stored":2298988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":140613,"internal_metadata":714857147},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738741,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":833548,"kb_used_data":4792,"kb_used_omap":115,"kb_used_meta":828620,"kb_avail":93538292,"statfs":{"total":96636764160,"available":95783211008,"internally_reserved":0,"allocated":4907008,"data_stored":2882754,"data_compressed":5893,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":118385,"internal_metadata":848507279},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738742,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1026312,"kb_used_data":4788,"kb_used_omap":140,"kb_used_meta":1021363,"kb_avail":93345528,"statfs":{"total":96636764160,"available":95585820672,"internally_reserved":0,"allocated":4902912,"data_stored":2882735,"data_compressed":5893,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":143603,"internal_metadata":1045876493},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5893,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5893,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T06:58:52.242 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-09-12T06:58:52.525 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:58:52.526 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":411,"fsid":"cccfff5c-d879-49a4-9cf1-b06db11d0b7e","created":"2024-09-12T06:27:54.820385+0000","modified":"2024-09-12T06:58:46.106702+0000","last_up_change":"2024-09-12T06:28:04.331593+0000","last_in_change":"2024-09-12T06:27:55.656795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":3,"require_min_compat_client":"octopus","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-12T06:28:04.463680+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-12T06:28:09.975795+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cdacbd5a-efce-4370-b990-b1e75740a4cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":2673152699},{"type":"v1","addr":"172.21.15.152:6817","nonce":2673152699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":2673152699},{"type":"v1","addr":"172.21.15.152:6819","nonce":2673152699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":2673152699},{"type":"v1","addr":"172.21.15.152:6823","nonce":2673152699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":2673152699},{"type":"v1","addr":"172.21.15.152:6821","nonce":2673152699}]},"public_addr":"172.21.15.152:6817/2673152699","cluster_addr":"172.21.15.152:6819/2673152699","heartbeat_back_addr":"172.21.15.152:6823/2673152699","heartbeat_front_addr":"172.21.15.152:6821/2673152699","state":["exists","up"]},{"osd":1,"uuid":"9019ee56-f19d-443f-99a7-7e04ef9915b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":1156225285},{"type":"v1","addr":"172.21.15.152:6809","nonce":1156225285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":1156225285},{"type":"v1","addr":"172.21.15.152:6811","nonce":1156225285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":1156225285},{"type":"v1","addr":"172.21.15.152:6815","nonce":1156225285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":1156225285},{"type":"v1","addr":"172.21.15.152:6813","nonce":1156225285}]},"public_addr":"172.21.15.152:6809/1156225285","cluster_addr":"172.21.15.152:6811/1156225285","heartbeat_back_addr":"172.21.15.152:6815/1156225285","heartbeat_front_addr":"172.21.15.152:6813/1156225285","state":["exists","up"]},{"osd":2,"uuid":"47b01d4d-e415-4491-a522-9513d1569287","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":2164790219},{"type":"v1","addr":"172.21.15.152:6801","nonce":2164790219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":2164790219},{"type":"v1","addr":"172.21.15.152:6803","nonce":2164790219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":2164790219},{"type":"v1","addr":"172.21.15.152:6807","nonce":2164790219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":2164790219},{"type":"v1","addr":"172.21.15.152:6805","nonce":2164790219}]},"public_addr":"172.21.15.152:6801/2164790219","cluster_addr":"172.21.15.152:6803/2164790219","heartbeat_back_addr":"172.21.15.152:6807/2164790219","heartbeat_front_addr":"172.21.15.152:6805/2164790219","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T06:28:01.930599+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T06:28:01.728162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T06:28:01.719714+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.152:0/222658089":"2024-09-12T07:58:40.339668+0000","172.21.15.152:0/1719428192":"2024-09-12T07:57:20.217333+0000","172.21.15.152:0/2155210955":"2024-09-12T07:55:17.258371+0000","172.21.15.152:0/1738048898":"2024-09-12T07:53:55.618609+0000","172.21.15.152:0/488149736":"2024-09-12T07:52:37.472013+0000","172.21.15.152:0/2723805928":"2024-09-12T07:50:47.443567+0000","172.21.15.152:0/3036446102":"2024-09-12T07:46:07.018635+0000","172.21.15.152:0/2265791919":"2024-09-12T07:45:38.709003+0000","172.21.15.152:0/2425621685":"2024-09-12T07:47:32.194747+0000","172.21.15.152:0/172419204":"2024-09-12T07:44:11.779615+0000","172.21.15.152:0/3530307948":"2024-09-12T07:48:54.364956+0000","172.21.15.152:0/21724814":"2024-09-12T07:46:59.991396+0000","172.21.15.152:0/3099155391":"2024-09-12T07:44:42.970933+0000","172.21.15.152:0/1430551510":"2024-09-12T07:46:48.714432+0000","172.21.15.152:0/3974301830":"2024-09-12T07:43:41.668630+0000","172.21.15.152:0/863186092":"2024-09-12T07:42:18.098685+0000","172.21.15.152:0/233534883":"2024-09-12T07:42:13.879662+0000","172.21.15.152:0/1201142280":"2024-09-12T07:50:07.699799+0000","172.21.15.152:0/369432838":"2024-09-12T07:31:48.047872+0000","172.21.15.152:0/3824019498":"2024-09-12T07:48:48.080883+0000","172.21.15.152:0/231937855":"2024-09-12T07:56:00.187374+0000","172.21.15.152:0/1050623528":"2024-09-12T07:41:35.925709+0000","172.21.15.152:0/3407836109":"2024-09-12T07:31:04.280458+0000","172.21.15.152:0/1544735163":"2024-09-12T07:50:14.012303+0000","172.21.15.152:0/775846538":"2024-09-12T07:35:33.405962+0000","172.21.15.152:0/3259234428":"2024-09-12T07:41:45.572522+0000","172.21.15.152:0/3160248264":"2024-09-12T07:46:54.812153+0000","172.21.15.152:0/2060810056":"2024-09-12T07:40:06.162284+0000","172.21.15.152:0/1334409024":"2024-09-12T07:46:44.045024+0000","172.21.15.152:0/2638046668":"2024-09-12T07:33:18.749414+0000","172.21.15.152:0/2460173709":"2024-09-12T07:34:01.852941+0000","172.21.15.152:0/3705705001":"2024-09-12T07:46:38.565649+0000","172.21.15.152:0/1325714198":"2024-09-12T07:32:36.185508+0000","172.21.15.152:0/2619275542":"2024-09-12T07:34:46.937826+0000","172.21.15.152:0/3450652672":"2024-09-12T07:56:36.409284+0000","172.21.15.152:0/4014429176":"2024-09-12T07:41:39.085630+0000","172.21.15.152:0/4197368466":"2024-09-12T07:36:25.042614+0000","172.21.15.152:0/1946299969":"2024-09-12T07:43:04.273513+0000","172.21.15.152:0/1412616417":"2024-09-12T07:47:25.793662+0000","172.21.15.152:0/2704108747":"2024-09-12T07:47:06.834833+0000","172.21.15.152:0/3181251308":"2024-09-12T07:37:22.947629+0000","172.21.15.152:0/1131214487":"2024-09-12T07:40:26.192891+0000","172.21.15.152:0/175340882":"2024-09-12T07:47:16.805689+0000","172.21.15.152:0/3738643920":"2024-09-12T07:47:36.839698+0000","172.21.15.152:0/314014393":"2024-09-12T07:47:42.086288+0000","172.21.15.152:0/2419126044":"2024-09-12T07:53:21.303820+0000","172.21.15.152:0/129583471":"2024-09-12T07:47:46.913035+0000","172.21.15.152:0/832024599":"2024-09-12T07:47:53.128667+0000","172.21.15.152:0/3992633193":"2024-09-12T07:48:21.798539+0000","172.21.15.152:0/3905057433":"2024-09-12T07:48:35.971200+0000","172.21.15.152:0/2493527828":"2024-09-12T07:48:14.922776+0000","172.21.15.152:0/2763807648":"2024-09-12T07:49:19.137278+0000","172.21.15.152:0/3971678597":"2024-09-12T07:50:39.857841+0000","172.21.15.152:0/1470009377":"2024-09-12T07:54:39.339058+0000","172.21.15.152:0/506129400":"2024-09-12T07:43:47.047406+0000","172.21.15.152:0/1607978993":"2024-09-12T07:48:42.456964+0000","172.21.15.152:0/1527590262":"2024-09-12T07:49:03.307065+0000","172.21.15.152:0/1746962075":"2024-09-12T07:49:33.996355+0000","172.21.15.152:0/1803419079":"2024-09-12T07:47:57.935351+0000","172.21.15.152:0/603323629":"2024-09-12T07:49:08.483029+0000","172.21.15.152:0/91705903":"2024-09-12T07:49:43.867125+0000","172.21.15.152:0/182078056":"2024-09-12T07:49:13.323443+0000","172.21.15.152:0/583314613":"2024-09-12T07:30:22.033649+0000","172.21.15.152:0/335796372":"2024-09-12T07:49:38.615266+0000","172.21.15.152:0/1789373293":"2024-09-12T07:49:52.860834+0000","172.21.15.152:0/4004070614":"2024-09-12T07:58:02.820642+0000","172.21.15.152:0/4069774543":"2024-09-12T07:49:57.694829+0000","172.21.15.152:0/2319608953":"2024-09-12T07:50:24.992869+0000","172.21.15.152:0/468075539":"2024-09-12T07:50:19.805403+0000","172.21.15.152:0/1037584147":"2024-09-12T07:50:30.794717+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-12T06:58:53.539 INFO:tasks.ceph:Scrubbing cluster2.osd.0 2024-09-12T06:58:53.539 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-12T06:58:53.676 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-12T06:58:53.676 INFO:teuthology.orchestra.run.smithi152.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-12T06:58:53.676 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-12T06:58:53.687 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 0 2024-09-12T06:58:53.969 INFO:teuthology.orchestra.run.smithi152.stderr:instructed osd(s) 0 to deep-scrub 2024-09-12T06:58:53.982 INFO:tasks.ceph:Scrubbing cluster2.osd.1 2024-09-12T06:58:53.982 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-12T06:58:54.106 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-12T06:58:54.106 INFO:teuthology.orchestra.run.smithi152.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-12T06:58:54.106 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-12T06:58:54.116 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 1 2024-09-12T06:58:54.402 INFO:teuthology.orchestra.run.smithi152.stderr:instructed osd(s) 1 to deep-scrub 2024-09-12T06:58:54.415 INFO:tasks.ceph:Scrubbing cluster2.osd.2 2024-09-12T06:58:54.415 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-12T06:58:54.538 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-09-12T06:58:54.538 INFO:teuthology.orchestra.run.smithi152.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-12T06:58:54.538 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-09-12T06:58:54.549 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 2 2024-09-12T06:58:54.835 INFO:teuthology.orchestra.run.smithi152.stderr:instructed osd(s) 2 to deep-scrub 2024-09-12T06:58:54.849 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T06:58:55.131 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:58:55.132 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-12T06:58:55.143 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":1334,"stamp":"2024-09-12T06:58:54.680730+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":58,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2562896,"kb_used_data":14352,"kb_used_omap":393,"kb_used_meta":2548086,"kb_avail":280552624,"statfs":{"total":289910292480,"available":287285886976,"internally_reserved":0,"allocated":14696448,"data_stored":8077738,"data_compressed":11786,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":402601,"internal_metadata":2609240919},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19,"num_objects":-6,"num_object_clones":0,"num_object_copies":-12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6,"num_whiteouts":0,"num_read":-1880,"num_read_kb":-1869,"num_write":-127,"num_write_kb":-124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001950"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":826,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-12T06:58:47.398944+0000","last_change":"2024-09-12T06:28:13.359110+0000","last_active":"2024-09-12T06:58:47.398944+0000","last_peered":"2024-09-12T06:58:47.398944+0000","last_clean":"2024-09-12T06:58:47.398944+0000","last_became_active":"2024-09-12T06:28:11.358431+0000","last_became_peered":"2024-09-12T06:28:11.358431+0000","last_unstale":"2024-09-12T06:58:47.398944+0000","last_undegraded":"2024-09-12T06:58:47.398944+0000","last_fullsized":"2024-09-12T06:58:47.398944+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_clean_scrub_stamp":"2024-09-12T06:28:10.346475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:11:44.092579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089666099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":826,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-12T06:58:47.401592+0000","last_change":"2024-09-12T06:28:13.359336+0000","last_active":"2024-09-12T06:58:47.401592+0000","last_peered":"2024-09-12T06:58:47.401592+0000","last_clean":"2024-09-12T06:58:47.401592+0000","last_became_active":"2024-09-12T06:28:11.358327+0000","last_became_peered":"2024-09-12T06:28:11.358327+0000","last_unstale":"2024-09-12T06:58:47.401592+0000","last_undegraded":"2024-09-12T06:58:47.401592+0000","last_fullsized":"2024-09-12T06:58:47.401592+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_clean_scrub_stamp":"2024-09-12T06:28:10.346475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:08:25.715384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001063589,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":826,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-12T06:58:47.402005+0000","last_change":"2024-09-12T06:28:13.358717+0000","last_active":"2024-09-12T06:58:47.402005+0000","last_peered":"2024-09-12T06:58:47.402005+0000","last_clean":"2024-09-12T06:58:47.402005+0000","last_became_active":"2024-09-12T06:28:11.358302+0000","last_became_peered":"2024-09-12T06:28:11.358302+0000","last_unstale":"2024-09-12T06:58:47.402005+0000","last_undegraded":"2024-09-12T06:58:47.402005+0000","last_fullsized":"2024-09-12T06:58:47.402005+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_clean_scrub_stamp":"2024-09-12T06:28:10.346475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T17:13:31.101064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055283899999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":826,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-12T06:58:47.399482+0000","last_change":"2024-09-12T06:28:13.358819+0000","last_active":"2024-09-12T06:58:47.399482+0000","last_peered":"2024-09-12T06:58:47.399482+0000","last_clean":"2024-09-12T06:58:47.399482+0000","last_became_active":"2024-09-12T06:28:11.357578+0000","last_became_peered":"2024-09-12T06:28:11.357578+0000","last_unstale":"2024-09-12T06:58:47.399482+0000","last_undegraded":"2024-09-12T06:58:47.399482+0000","last_fullsized":"2024-09-12T06:58:47.399482+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_clean_scrub_stamp":"2024-09-12T06:28:10.346475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T08:54:00.421871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000609568,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":828,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-12T06:58:46.112984+0000","last_change":"2024-09-12T06:28:13.364918+0000","last_active":"2024-09-12T06:58:46.112984+0000","last_peered":"2024-09-12T06:58:46.112984+0000","last_clean":"2024-09-12T06:58:46.112984+0000","last_became_active":"2024-09-12T06:28:11.357233+0000","last_became_peered":"2024-09-12T06:28:11.357233+0000","last_unstale":"2024-09-12T06:58:46.112984+0000","last_undegraded":"2024-09-12T06:58:46.112984+0000","last_fullsized":"2024-09-12T06:58:46.112984+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_clean_scrub_stamp":"2024-09-12T06:28:10.346475+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:37:00.884436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012297149999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":826,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-12T06:58:47.878327+0000","last_change":"2024-09-12T06:28:13.473086+0000","last_active":"2024-09-12T06:58:47.878327+0000","last_peered":"2024-09-12T06:58:47.878327+0000","last_clean":"2024-09-12T06:58:47.878327+0000","last_became_active":"2024-09-12T06:28:11.357446+0000","last_became_peered":"2024-09-12T06:28:11.357446+0000","last_unstale":"2024-09-12T06:58:47.878327+0000","last_undegraded":"2024-09-12T06:58:47.878327+0000","last_fullsized":"2024-09-12T06:58:47.878327+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_clean_scrub_stamp":"2024-09-12T06:28:10.346475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T16:51:04.766222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041499400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":827,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-12T06:58:47.878869+0000","last_change":"2024-09-12T06:28:13.473086+0000","last_active":"2024-09-12T06:58:47.878869+0000","last_peered":"2024-09-12T06:58:47.878869+0000","last_clean":"2024-09-12T06:58:47.878869+0000","last_became_active":"2024-09-12T06:28:11.359644+0000","last_became_peered":"2024-09-12T06:28:11.359644+0000","last_unstale":"2024-09-12T06:58:47.878869+0000","last_undegraded":"2024-09-12T06:58:47.878869+0000","last_fullsized":"2024-09-12T06:58:47.878869+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_clean_scrub_stamp":"2024-09-12T06:28:10.346475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T16:48:40.580401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000404719,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":827,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-12T06:58:47.400970+0000","last_change":"2024-09-12T06:28:13.359050+0000","last_active":"2024-09-12T06:58:47.400970+0000","last_peered":"2024-09-12T06:58:47.400970+0000","last_clean":"2024-09-12T06:58:47.400970+0000","last_became_active":"2024-09-12T06:28:11.358377+0000","last_became_peered":"2024-09-12T06:28:11.358377+0000","last_unstale":"2024-09-12T06:58:47.400970+0000","last_undegraded":"2024-09-12T06:58:47.400970+0000","last_fullsized":"2024-09-12T06:58:47.400970+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:28:10.346475+0000","last_clean_scrub_stamp":"2024-09-12T06:28:10.346475+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:05:39.794241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084816700000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'52","reported_seq":895,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-12T06:58:47.396828+0000","last_change":"2024-09-12T06:28:06.474810+0000","last_active":"2024-09-12T06:58:47.396828+0000","last_peered":"2024-09-12T06:58:47.396828+0000","last_clean":"2024-09-12T06:58:47.396828+0000","last_became_active":"2024-09-12T06:28:06.473701+0000","last_became_peered":"2024-09-12T06:28:06.473701+0000","last_unstale":"2024-09-12T06:58:47.396828+0000","last_undegraded":"2024-09-12T06:58:47.396828+0000","last_fullsized":"2024-09-12T06:58:47.396828+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:28:05.332439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:28:05.332439+0000","last_clean_scrub_stamp":"2024-09-12T06:28:05.332439+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T16:09:15.324127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11786,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738743,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703004,"kb_used_data":4744,"kb_used_omap":137,"kb_used_meta":698102,"kb_avail":93668836,"statfs":{"total":96636764160,"available":95916888064,"internally_reserved":0,"allocated":4857856,"data_stored":2298988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":140613,"internal_metadata":714857147},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738742,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":833560,"kb_used_data":4804,"kb_used_omap":115,"kb_used_meta":828620,"kb_avail":93538280,"statfs":{"total":96636764160,"available":95783198720,"internally_reserved":0,"allocated":4919296,"data_stored":2889375,"data_compressed":5893,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":118385,"internal_metadata":848507279},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738743,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1026332,"kb_used_data":4804,"kb_used_omap":140,"kb_used_meta":1021363,"kb_avail":93345508,"statfs":{"total":96636764160,"available":95585800192,"internally_reserved":0,"allocated":4919296,"data_stored":2889375,"data_compressed":5893,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":143603,"internal_metadata":1045876493},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5893,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5893,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T06:58:55.147 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-12T06:28:10.346475+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=6, tm_min=28, tm_sec=10, tm_wday=3, tm_yday=256, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=6, tm_min=58, tm_sec=52, tm_wday=3, tm_yday=256, tm_isdst=0) 2024-09-12T06:58:55.148 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-12T06:28:10.346475+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=6, tm_min=28, tm_sec=10, tm_wday=3, tm_yday=256, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=6, tm_min=58, tm_sec=52, tm_wday=3, tm_yday=256, tm_isdst=0) 2024-09-12T06:58:55.148 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-12T06:28:10.346475+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=6, tm_min=28, tm_sec=10, tm_wday=3, tm_yday=256, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=6, tm_min=58, tm_sec=52, tm_wday=3, tm_yday=256, tm_isdst=0) 2024-09-12T06:58:55.148 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-12T06:28:10.346475+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=6, tm_min=28, tm_sec=10, tm_wday=3, tm_yday=256, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=6, tm_min=58, tm_sec=52, tm_wday=3, tm_yday=256, tm_isdst=0) 2024-09-12T06:58:55.148 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-12T06:28:10.346475+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=6, tm_min=28, tm_sec=10, tm_wday=3, tm_yday=256, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=6, tm_min=58, tm_sec=52, tm_wday=3, tm_yday=256, tm_isdst=0) 2024-09-12T06:58:55.148 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-12T06:28:10.346475+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=6, tm_min=28, tm_sec=10, tm_wday=3, tm_yday=256, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=6, tm_min=58, tm_sec=52, tm_wday=3, tm_yday=256, tm_isdst=0) 2024-09-12T06:58:55.148 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-12T06:28:10.346475+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=6, tm_min=28, tm_sec=10, tm_wday=3, tm_yday=256, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=6, tm_min=58, tm_sec=52, tm_wday=3, tm_yday=256, tm_isdst=0) 2024-09-12T06:58:55.148 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-12T06:28:10.346475+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=6, tm_min=28, tm_sec=10, tm_wday=3, tm_yday=256, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=6, tm_min=58, tm_sec=52, tm_wday=3, tm_yday=256, tm_isdst=0) 2024-09-12T06:58:55.148 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-12T06:28:05.332439+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=6, tm_min=28, tm_sec=5, tm_wday=3, tm_yday=256, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=6, tm_min=58, tm_sec=52, tm_wday=3, tm_yday=256, tm_isdst=0) 2024-09-12T06:58:55.148 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-12T06:59:15.150 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T06:59:15.445 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-09-12T06:59:15.445 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-09-12T06:59:15.457 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":1344,"stamp":"2024-09-12T06:59:14.684568+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2562880,"kb_used_data":14352,"kb_used_omap":393,"kb_used_meta":2548086,"kb_avail":280552640,"statfs":{"total":289910292480,"available":287285903360,"internally_reserved":0,"allocated":14696448,"data_stored":8077738,"data_compressed":11786,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":402601,"internal_metadata":2609240919},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002386"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":834,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-12T06:59:00.350142+0000","last_change":"2024-09-12T06:59:00.350142+0000","last_active":"2024-09-12T06:59:00.350142+0000","last_peered":"2024-09-12T06:59:00.350142+0000","last_clean":"2024-09-12T06:59:00.350142+0000","last_became_active":"2024-09-12T06:28:11.358431+0000","last_became_peered":"2024-09-12T06:28:11.358431+0000","last_unstale":"2024-09-12T06:59:00.350142+0000","last_undegraded":"2024-09-12T06:59:00.350142+0000","last_fullsized":"2024-09-12T06:59:00.350142+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:59:00.350052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:59:00.350052+0000","last_clean_scrub_stamp":"2024-09-12T06:59:00.350052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:17:04.045663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089666099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":834,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-12T06:58:59.330181+0000","last_change":"2024-09-12T06:58:59.330181+0000","last_active":"2024-09-12T06:58:59.330181+0000","last_peered":"2024-09-12T06:58:59.330181+0000","last_clean":"2024-09-12T06:58:59.330181+0000","last_became_active":"2024-09-12T06:28:11.358327+0000","last_became_peered":"2024-09-12T06:28:11.358327+0000","last_unstale":"2024-09-12T06:58:59.330181+0000","last_undegraded":"2024-09-12T06:58:59.330181+0000","last_fullsized":"2024-09-12T06:58:59.330181+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:58:59.330065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:58:59.330065+0000","last_clean_scrub_stamp":"2024-09-12T06:58:59.330065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:24:50.146597+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001063589,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":834,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-12T06:58:57.372167+0000","last_change":"2024-09-12T06:58:57.372167+0000","last_active":"2024-09-12T06:58:57.372167+0000","last_peered":"2024-09-12T06:58:57.372167+0000","last_clean":"2024-09-12T06:58:57.372167+0000","last_became_active":"2024-09-12T06:28:11.358302+0000","last_became_peered":"2024-09-12T06:28:11.358302+0000","last_unstale":"2024-09-12T06:58:57.372167+0000","last_undegraded":"2024-09-12T06:58:57.372167+0000","last_fullsized":"2024-09-12T06:58:57.372167+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:58:57.372077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:58:57.372077+0000","last_clean_scrub_stamp":"2024-09-12T06:58:57.372077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:08:39.650171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055283899999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":834,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-12T06:58:55.356833+0000","last_change":"2024-09-12T06:58:55.356833+0000","last_active":"2024-09-12T06:58:55.356833+0000","last_peered":"2024-09-12T06:58:55.356833+0000","last_clean":"2024-09-12T06:58:55.356833+0000","last_became_active":"2024-09-12T06:28:11.357578+0000","last_became_peered":"2024-09-12T06:28:11.357578+0000","last_unstale":"2024-09-12T06:58:55.356833+0000","last_undegraded":"2024-09-12T06:58:55.356833+0000","last_fullsized":"2024-09-12T06:58:55.356833+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:58:55.356752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:58:55.356752+0000","last_clean_scrub_stamp":"2024-09-12T06:58:55.356752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:29:14.652990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000609568,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"411'3","reported_seq":837,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-12T06:58:54.039832+0000","last_change":"2024-09-12T06:58:54.039832+0000","last_active":"2024-09-12T06:58:54.039832+0000","last_peered":"2024-09-12T06:58:54.039832+0000","last_clean":"2024-09-12T06:58:54.039832+0000","last_became_active":"2024-09-12T06:28:11.357233+0000","last_became_peered":"2024-09-12T06:28:11.357233+0000","last_unstale":"2024-09-12T06:58:54.039832+0000","last_undegraded":"2024-09-12T06:58:54.039832+0000","last_fullsized":"2024-09-12T06:58:54.039832+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"411'3","last_scrub_stamp":"2024-09-12T06:58:54.039766+0000","last_deep_scrub":"411'3","last_deep_scrub_stamp":"2024-09-12T06:58:54.039766+0000","last_clean_scrub_stamp":"2024-09-12T06:58:54.039766+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T16:09:14.868781+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0012297149999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":834,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-12T06:58:56.705951+0000","last_change":"2024-09-12T06:58:56.705951+0000","last_active":"2024-09-12T06:58:56.705951+0000","last_peered":"2024-09-12T06:58:56.705951+0000","last_clean":"2024-09-12T06:58:56.705951+0000","last_became_active":"2024-09-12T06:28:11.357446+0000","last_became_peered":"2024-09-12T06:28:11.357446+0000","last_unstale":"2024-09-12T06:58:56.705951+0000","last_undegraded":"2024-09-12T06:58:56.705951+0000","last_fullsized":"2024-09-12T06:58:56.705951+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:58:56.705855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:58:56.705855+0000","last_clean_scrub_stamp":"2024-09-12T06:58:56.705855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:07:59.506723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041499400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":835,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-12T06:58:55.671351+0000","last_change":"2024-09-12T06:58:55.671351+0000","last_active":"2024-09-12T06:58:55.671351+0000","last_peered":"2024-09-12T06:58:55.671351+0000","last_clean":"2024-09-12T06:58:55.671351+0000","last_became_active":"2024-09-12T06:28:11.359644+0000","last_became_peered":"2024-09-12T06:28:11.359644+0000","last_unstale":"2024-09-12T06:58:55.671351+0000","last_undegraded":"2024-09-12T06:58:55.671351+0000","last_fullsized":"2024-09-12T06:58:55.671351+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:58:55.671283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:58:55.671283+0000","last_clean_scrub_stamp":"2024-09-12T06:58:55.671283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:44:12.982707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000404719,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"411'2","reported_seq":836,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-12T06:58:56.383722+0000","last_change":"2024-09-12T06:58:56.383722+0000","last_active":"2024-09-12T06:58:56.383722+0000","last_peered":"2024-09-12T06:58:56.383722+0000","last_clean":"2024-09-12T06:58:56.383722+0000","last_became_active":"2024-09-12T06:28:11.358377+0000","last_became_peered":"2024-09-12T06:28:11.358377+0000","last_unstale":"2024-09-12T06:58:56.383722+0000","last_undegraded":"2024-09-12T06:58:56.383722+0000","last_fullsized":"2024-09-12T06:58:56.383722+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"411'2","last_scrub_stamp":"2024-09-12T06:58:56.383639+0000","last_deep_scrub":"411'2","last_deep_scrub_stamp":"2024-09-12T06:58:56.383639+0000","last_clean_scrub_stamp":"2024-09-12T06:58:56.383639+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:37:20.530991+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00084816700000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"411'54","reported_seq":905,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-12T06:58:58.343261+0000","last_change":"2024-09-12T06:58:58.343261+0000","last_active":"2024-09-12T06:58:58.343261+0000","last_peered":"2024-09-12T06:58:58.343261+0000","last_clean":"2024-09-12T06:58:58.343261+0000","last_became_active":"2024-09-12T06:28:06.473701+0000","last_became_peered":"2024-09-12T06:28:06.473701+0000","last_unstale":"2024-09-12T06:58:58.343261+0000","last_undegraded":"2024-09-12T06:58:58.343261+0000","last_fullsized":"2024-09-12T06:58:58.343261+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"411'54","last_scrub_stamp":"2024-09-12T06:58:58.343162+0000","last_deep_scrub":"411'54","last_deep_scrub_stamp":"2024-09-12T06:58:58.343162+0000","last_clean_scrub_stamp":"2024-09-12T06:58:58.343162+0000","objects_scrubbed":2,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T17:09:32.467970+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11786,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":54,"ondisk_log_size":54,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738747,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703000,"kb_used_data":4744,"kb_used_omap":137,"kb_used_meta":698102,"kb_avail":93668840,"statfs":{"total":96636764160,"available":95916892160,"internally_reserved":0,"allocated":4857856,"data_stored":2298988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":140613,"internal_metadata":714857147},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738746,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":833556,"kb_used_data":4804,"kb_used_omap":115,"kb_used_meta":828620,"kb_avail":93538284,"statfs":{"total":96636764160,"available":95783202816,"internally_reserved":0,"allocated":4919296,"data_stored":2889375,"data_compressed":5893,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":118385,"internal_metadata":848507279},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738747,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1026324,"kb_used_data":4804,"kb_used_omap":140,"kb_used_meta":1021363,"kb_avail":93345516,"statfs":{"total":96636764160,"available":95585808384,"internally_reserved":0,"allocated":4919296,"data_stored":2889375,"data_compressed":5893,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":143603,"internal_metadata":1045876493},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5893,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5893,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T06:59:15.459 DEBUG:teuthology.orchestra.run.smithi152:> sudo ceph --cluster cluster2 config set global mon_health_to_clog false 2024-09-12T06:59:15.756 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-12T06:59:15.756 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-12T06:59:15.756 DEBUG:tasks.ceph.cluster2.osd.0:waiting for process to exit 2024-09-12T06:59:15.756 INFO:teuthology.orchestra.run:waiting for 300 2024-09-12T06:59:15.822 INFO:tasks.ceph.cluster2.osd.0:Stopped 2024-09-12T06:59:15.822 DEBUG:tasks.ceph.cluster2.osd.1:waiting for process to exit 2024-09-12T06:59:15.822 INFO:teuthology.orchestra.run:waiting for 300 2024-09-12T06:59:15.899 INFO:tasks.ceph.cluster2.osd.1:Stopped 2024-09-12T06:59:15.900 DEBUG:tasks.ceph.cluster2.osd.2:waiting for process to exit 2024-09-12T06:59:15.900 INFO:teuthology.orchestra.run:waiting for 300 2024-09-12T06:59:15.971 INFO:tasks.ceph.cluster2.osd.2:Stopped 2024-09-12T06:59:15.971 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-12T06:59:15.971 DEBUG:tasks.ceph.cluster2.mgr.x:waiting for process to exit 2024-09-12T06:59:15.971 INFO:teuthology.orchestra.run:waiting for 300 2024-09-12T06:59:16.015 INFO:tasks.ceph.cluster2.mgr.x:Stopped 2024-09-12T06:59:16.015 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-12T06:59:16.016 DEBUG:tasks.ceph.cluster2.mon.a:waiting for process to exit 2024-09-12T06:59:16.016 INFO:teuthology.orchestra.run:waiting for 300 2024-09-12T06:59:16.028 INFO:tasks.ceph.cluster2.mon.a:Stopped 2024-09-12T06:59:16.028 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-12T06:59:16.028 DEBUG:teuthology.orchestra.run.smithi152:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-09-12T06:59:16.161 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-0 on ubuntu@smithi152.front.sepia.ceph.com 2024-09-12T06:59:16.161 DEBUG:teuthology.orchestra.run.smithi152:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-0 2024-09-12T06:59:16.422 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-1 on ubuntu@smithi152.front.sepia.ceph.com 2024-09-12T06:59:16.422 DEBUG:teuthology.orchestra.run.smithi152:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-1 2024-09-12T06:59:16.560 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-2 on ubuntu@smithi152.front.sepia.ceph.com 2024-09-12T06:59:16.561 DEBUG:teuthology.orchestra.run.smithi152:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-2 2024-09-12T06:59:16.692 INFO:tasks.ceph:Archiving mon data... 2024-09-12T06:59:16.693 DEBUG:teuthology.misc:Transferring archived files from smithi152:/var/lib/ceph/mon/cluster2-a to /home/teuthworker/archive/dis-2024-09-12_04:46:10-rbd-wip-dis-testing-distro-default-smithi/7902364/data/cluster2.mon.a.tgz 2024-09-12T06:59:16.694 DEBUG:teuthology.orchestra.run.smithi152:> mktemp 2024-09-12T06:59:16.713 INFO:teuthology.orchestra.run.smithi152.stdout:/tmp/tmp.a8eQyuN2cs 2024-09-12T06:59:16.713 DEBUG:teuthology.orchestra.run.smithi152:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster2-a -- . > /tmp/tmp.a8eQyuN2cs 2024-09-12T06:59:16.905 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod 0666 /tmp/tmp.a8eQyuN2cs 2024-09-12T06:59:17.014 DEBUG:teuthology.orchestra.remote:smithi152:/tmp/tmp.a8eQyuN2cs is 476KB 2024-09-12T06:59:17.087 DEBUG:teuthology.orchestra.run.smithi152:> rm -fr /tmp/tmp.a8eQyuN2cs 2024-09-12T06:59:17.103 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-12T06:59:17.103 DEBUG:teuthology.orchestra.run.smithi096:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2024-09-12T06:59:17.106 DEBUG:teuthology.orchestra.run.smithi152:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2024-09-12T06:59:17.174 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-12T06:59:17.183 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2024-09-12T06:59:17.184 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-12T06:59:17.717 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:59:17.717 INFO:teuthology.orchestra.run.smithi096.stderr:dumped all 2024-09-12T06:59:17.729 INFO:teuthology.orchestra.run.smithi096.stdout:{"pg_ready":true,"pg_map":{"version":1291,"stamp":"2024-09-12T06:59:16.500287+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4536768,"kb_used_data":8256,"kb_used_omap":462,"kb_used_meta":4527985,"kb_avail":278578752,"statfs":{"total":289910292480,"available":285264642048,"internally_reserved":0,"allocated":8454144,"data_stored":5575555,"data_compressed":11786,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":473703,"internal_metadata":4636657049},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002085"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":678,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-12T06:58:44.978639+0000","last_change":"2024-09-12T06:27:34.239517+0000","last_active":"2024-09-12T06:58:44.978639+0000","last_peered":"2024-09-12T06:58:44.978639+0000","last_clean":"2024-09-12T06:58:44.978639+0000","last_became_active":"2024-09-12T06:27:32.239188+0000","last_became_peered":"2024-09-12T06:27:32.239188+0000","last_unstale":"2024-09-12T06:58:44.978639+0000","last_undegraded":"2024-09-12T06:58:44.978639+0000","last_fullsized":"2024-09-12T06:58:44.978639+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_clean_scrub_stamp":"2024-09-12T06:27:31.224809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T15:03:24.767390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011638639999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":678,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-12T06:58:44.977748+0000","last_change":"2024-09-12T06:27:34.239133+0000","last_active":"2024-09-12T06:58:44.977748+0000","last_peered":"2024-09-12T06:58:44.977748+0000","last_clean":"2024-09-12T06:58:44.977748+0000","last_became_active":"2024-09-12T06:27:32.239159+0000","last_became_peered":"2024-09-12T06:27:32.239159+0000","last_unstale":"2024-09-12T06:58:44.977748+0000","last_undegraded":"2024-09-12T06:58:44.977748+0000","last_fullsized":"2024-09-12T06:58:44.977748+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_clean_scrub_stamp":"2024-09-12T06:27:31.224809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:25:39.727415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079824799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":679,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-12T06:58:44.979807+0000","last_change":"2024-09-12T06:27:34.238743+0000","last_active":"2024-09-12T06:58:44.979807+0000","last_peered":"2024-09-12T06:58:44.979807+0000","last_clean":"2024-09-12T06:58:44.979807+0000","last_became_active":"2024-09-12T06:27:32.239133+0000","last_became_peered":"2024-09-12T06:27:32.239133+0000","last_unstale":"2024-09-12T06:58:44.979807+0000","last_undegraded":"2024-09-12T06:58:44.979807+0000","last_fullsized":"2024-09-12T06:58:44.979807+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_clean_scrub_stamp":"2024-09-12T06:27:31.224809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:11:59.861727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050645400000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":679,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-12T06:58:44.975906+0000","last_change":"2024-09-12T06:27:34.238755+0000","last_active":"2024-09-12T06:58:44.975906+0000","last_peered":"2024-09-12T06:58:44.975906+0000","last_clean":"2024-09-12T06:58:44.975906+0000","last_became_active":"2024-09-12T06:27:32.239390+0000","last_became_peered":"2024-09-12T06:27:32.239390+0000","last_unstale":"2024-09-12T06:58:44.975906+0000","last_undegraded":"2024-09-12T06:58:44.975906+0000","last_fullsized":"2024-09-12T06:58:44.975906+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_clean_scrub_stamp":"2024-09-12T06:27:31.224809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:18:29.532273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050204899999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":680,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-12T06:58:44.979328+0000","last_change":"2024-09-12T06:27:34.242205+0000","last_active":"2024-09-12T06:58:44.979328+0000","last_peered":"2024-09-12T06:58:44.979328+0000","last_clean":"2024-09-12T06:58:44.979328+0000","last_became_active":"2024-09-12T06:27:32.240428+0000","last_became_peered":"2024-09-12T06:27:32.240428+0000","last_unstale":"2024-09-12T06:58:44.979328+0000","last_undegraded":"2024-09-12T06:58:44.979328+0000","last_fullsized":"2024-09-12T06:58:44.979328+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_clean_scrub_stamp":"2024-09-12T06:27:31.224809+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:37:32.701549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095364099999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":679,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-12T06:58:45.695088+0000","last_change":"2024-09-12T06:27:34.508520+0000","last_active":"2024-09-12T06:58:45.695088+0000","last_peered":"2024-09-12T06:58:45.695088+0000","last_clean":"2024-09-12T06:58:45.695088+0000","last_became_active":"2024-09-12T06:27:32.239409+0000","last_became_peered":"2024-09-12T06:27:32.239409+0000","last_unstale":"2024-09-12T06:58:45.695088+0000","last_undegraded":"2024-09-12T06:58:45.695088+0000","last_fullsized":"2024-09-12T06:58:45.695088+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_clean_scrub_stamp":"2024-09-12T06:27:31.224809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T10:21:29.326121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045469599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":679,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-12T06:58:45.694715+0000","last_change":"2024-09-12T06:27:34.508464+0000","last_active":"2024-09-12T06:58:45.694715+0000","last_peered":"2024-09-12T06:58:45.694715+0000","last_clean":"2024-09-12T06:58:45.694715+0000","last_became_active":"2024-09-12T06:27:32.239718+0000","last_became_peered":"2024-09-12T06:27:32.239718+0000","last_unstale":"2024-09-12T06:58:45.694715+0000","last_undegraded":"2024-09-12T06:58:45.694715+0000","last_fullsized":"2024-09-12T06:58:45.694715+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_clean_scrub_stamp":"2024-09-12T06:27:31.224809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:21:30.104428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039987200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":680,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-12T06:58:44.977057+0000","last_change":"2024-09-12T06:27:34.239177+0000","last_active":"2024-09-12T06:58:44.977057+0000","last_peered":"2024-09-12T06:58:44.977057+0000","last_clean":"2024-09-12T06:58:44.977057+0000","last_became_active":"2024-09-12T06:27:32.239878+0000","last_became_peered":"2024-09-12T06:27:32.239878+0000","last_unstale":"2024-09-12T06:58:44.977057+0000","last_undegraded":"2024-09-12T06:58:44.977057+0000","last_fullsized":"2024-09-12T06:58:44.977057+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_clean_scrub_stamp":"2024-09-12T06:27:31.224809+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:11:21.471227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091624399999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'52","reported_seq":747,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-12T06:58:44.976209+0000","last_change":"2024-09-12T06:27:27.078058+0000","last_active":"2024-09-12T06:58:44.976209+0000","last_peered":"2024-09-12T06:58:44.976209+0000","last_clean":"2024-09-12T06:58:44.976209+0000","last_became_active":"2024-09-12T06:27:27.076932+0000","last_became_peered":"2024-09-12T06:27:27.076932+0000","last_unstale":"2024-09-12T06:58:44.976209+0000","last_undegraded":"2024-09-12T06:58:44.976209+0000","last_fullsized":"2024-09-12T06:58:44.976209+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:27:25.120999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:27:25.120999+0000","last_clean_scrub_stamp":"2024-09-12T06:27:25.120999+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:20:08.653460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11786,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":9,"seq":38654706050,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1307752,"kb_used_data":2772,"kb_used_omap":204,"kb_used_meta":1304755,"kb_avail":93064088,"statfs":{"total":96636764160,"available":95297626112,"internally_reserved":0,"allocated":2838528,"data_stored":2055314,"data_compressed":5893,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":209748,"internal_metadata":1336069292},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738754,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1436076,"kb_used_data":2712,"kb_used_omap":135,"kb_used_meta":1433208,"kb_avail":92935764,"statfs":{"total":96636764160,"available":95166222336,"internally_reserved":0,"allocated":2777088,"data_stored":1464927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":139153,"internal_metadata":1467605103},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738754,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1792940,"kb_used_data":2772,"kb_used_omap":121,"kb_used_meta":1790022,"kb_avail":92578900,"statfs":{"total":96636764160,"available":94800793600,"internally_reserved":0,"allocated":2838528,"data_stored":2055314,"data_compressed":5893,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":124802,"internal_metadata":1832982654},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5893,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5893,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T06:59:17.730 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-12T06:59:18.009 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:59:18.010 INFO:teuthology.orchestra.run.smithi096.stderr:dumped all 2024-09-12T06:59:18.022 INFO:teuthology.orchestra.run.smithi096.stdout:{"pg_ready":true,"pg_map":{"version":1291,"stamp":"2024-09-12T06:59:16.500287+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4536768,"kb_used_data":8256,"kb_used_omap":462,"kb_used_meta":4527985,"kb_avail":278578752,"statfs":{"total":289910292480,"available":285264642048,"internally_reserved":0,"allocated":8454144,"data_stored":5575555,"data_compressed":11786,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":473703,"internal_metadata":4636657049},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002085"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":678,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-12T06:58:44.978639+0000","last_change":"2024-09-12T06:27:34.239517+0000","last_active":"2024-09-12T06:58:44.978639+0000","last_peered":"2024-09-12T06:58:44.978639+0000","last_clean":"2024-09-12T06:58:44.978639+0000","last_became_active":"2024-09-12T06:27:32.239188+0000","last_became_peered":"2024-09-12T06:27:32.239188+0000","last_unstale":"2024-09-12T06:58:44.978639+0000","last_undegraded":"2024-09-12T06:58:44.978639+0000","last_fullsized":"2024-09-12T06:58:44.978639+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_clean_scrub_stamp":"2024-09-12T06:27:31.224809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T15:03:24.767390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011638639999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":678,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-12T06:58:44.977748+0000","last_change":"2024-09-12T06:27:34.239133+0000","last_active":"2024-09-12T06:58:44.977748+0000","last_peered":"2024-09-12T06:58:44.977748+0000","last_clean":"2024-09-12T06:58:44.977748+0000","last_became_active":"2024-09-12T06:27:32.239159+0000","last_became_peered":"2024-09-12T06:27:32.239159+0000","last_unstale":"2024-09-12T06:58:44.977748+0000","last_undegraded":"2024-09-12T06:58:44.977748+0000","last_fullsized":"2024-09-12T06:58:44.977748+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_clean_scrub_stamp":"2024-09-12T06:27:31.224809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:25:39.727415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079824799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":679,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-12T06:58:44.979807+0000","last_change":"2024-09-12T06:27:34.238743+0000","last_active":"2024-09-12T06:58:44.979807+0000","last_peered":"2024-09-12T06:58:44.979807+0000","last_clean":"2024-09-12T06:58:44.979807+0000","last_became_active":"2024-09-12T06:27:32.239133+0000","last_became_peered":"2024-09-12T06:27:32.239133+0000","last_unstale":"2024-09-12T06:58:44.979807+0000","last_undegraded":"2024-09-12T06:58:44.979807+0000","last_fullsized":"2024-09-12T06:58:44.979807+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_clean_scrub_stamp":"2024-09-12T06:27:31.224809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:11:59.861727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050645400000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":679,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-12T06:58:44.975906+0000","last_change":"2024-09-12T06:27:34.238755+0000","last_active":"2024-09-12T06:58:44.975906+0000","last_peered":"2024-09-12T06:58:44.975906+0000","last_clean":"2024-09-12T06:58:44.975906+0000","last_became_active":"2024-09-12T06:27:32.239390+0000","last_became_peered":"2024-09-12T06:27:32.239390+0000","last_unstale":"2024-09-12T06:58:44.975906+0000","last_undegraded":"2024-09-12T06:58:44.975906+0000","last_fullsized":"2024-09-12T06:58:44.975906+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_clean_scrub_stamp":"2024-09-12T06:27:31.224809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:18:29.532273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050204899999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":680,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-12T06:58:44.979328+0000","last_change":"2024-09-12T06:27:34.242205+0000","last_active":"2024-09-12T06:58:44.979328+0000","last_peered":"2024-09-12T06:58:44.979328+0000","last_clean":"2024-09-12T06:58:44.979328+0000","last_became_active":"2024-09-12T06:27:32.240428+0000","last_became_peered":"2024-09-12T06:27:32.240428+0000","last_unstale":"2024-09-12T06:58:44.979328+0000","last_undegraded":"2024-09-12T06:58:44.979328+0000","last_fullsized":"2024-09-12T06:58:44.979328+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_clean_scrub_stamp":"2024-09-12T06:27:31.224809+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:37:32.701549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095364099999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":679,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-12T06:58:45.695088+0000","last_change":"2024-09-12T06:27:34.508520+0000","last_active":"2024-09-12T06:58:45.695088+0000","last_peered":"2024-09-12T06:58:45.695088+0000","last_clean":"2024-09-12T06:58:45.695088+0000","last_became_active":"2024-09-12T06:27:32.239409+0000","last_became_peered":"2024-09-12T06:27:32.239409+0000","last_unstale":"2024-09-12T06:58:45.695088+0000","last_undegraded":"2024-09-12T06:58:45.695088+0000","last_fullsized":"2024-09-12T06:58:45.695088+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_clean_scrub_stamp":"2024-09-12T06:27:31.224809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T10:21:29.326121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045469599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":679,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-12T06:58:45.694715+0000","last_change":"2024-09-12T06:27:34.508464+0000","last_active":"2024-09-12T06:58:45.694715+0000","last_peered":"2024-09-12T06:58:45.694715+0000","last_clean":"2024-09-12T06:58:45.694715+0000","last_became_active":"2024-09-12T06:27:32.239718+0000","last_became_peered":"2024-09-12T06:27:32.239718+0000","last_unstale":"2024-09-12T06:58:45.694715+0000","last_undegraded":"2024-09-12T06:58:45.694715+0000","last_fullsized":"2024-09-12T06:58:45.694715+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_clean_scrub_stamp":"2024-09-12T06:27:31.224809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:21:30.104428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039987200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":680,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-12T06:58:44.977057+0000","last_change":"2024-09-12T06:27:34.239177+0000","last_active":"2024-09-12T06:58:44.977057+0000","last_peered":"2024-09-12T06:58:44.977057+0000","last_clean":"2024-09-12T06:58:44.977057+0000","last_became_active":"2024-09-12T06:27:32.239878+0000","last_became_peered":"2024-09-12T06:27:32.239878+0000","last_unstale":"2024-09-12T06:58:44.977057+0000","last_undegraded":"2024-09-12T06:58:44.977057+0000","last_fullsized":"2024-09-12T06:58:44.977057+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_clean_scrub_stamp":"2024-09-12T06:27:31.224809+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:11:21.471227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091624399999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'52","reported_seq":747,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-12T06:58:44.976209+0000","last_change":"2024-09-12T06:27:27.078058+0000","last_active":"2024-09-12T06:58:44.976209+0000","last_peered":"2024-09-12T06:58:44.976209+0000","last_clean":"2024-09-12T06:58:44.976209+0000","last_became_active":"2024-09-12T06:27:27.076932+0000","last_became_peered":"2024-09-12T06:27:27.076932+0000","last_unstale":"2024-09-12T06:58:44.976209+0000","last_undegraded":"2024-09-12T06:58:44.976209+0000","last_fullsized":"2024-09-12T06:58:44.976209+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:27:25.120999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:27:25.120999+0000","last_clean_scrub_stamp":"2024-09-12T06:27:25.120999+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:20:08.653460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11786,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":9,"seq":38654706050,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1307752,"kb_used_data":2772,"kb_used_omap":204,"kb_used_meta":1304755,"kb_avail":93064088,"statfs":{"total":96636764160,"available":95297626112,"internally_reserved":0,"allocated":2838528,"data_stored":2055314,"data_compressed":5893,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":209748,"internal_metadata":1336069292},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738754,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1436076,"kb_used_data":2712,"kb_used_omap":135,"kb_used_meta":1433208,"kb_avail":92935764,"statfs":{"total":96636764160,"available":95166222336,"internally_reserved":0,"allocated":2777088,"data_stored":1464927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":139153,"internal_metadata":1467605103},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738754,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1792940,"kb_used_data":2772,"kb_used_omap":121,"kb_used_meta":1790022,"kb_avail":92578900,"statfs":{"total":96636764160,"available":94800793600,"internally_reserved":0,"allocated":2838528,"data_stored":2055314,"data_compressed":5893,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":124802,"internal_metadata":1832982654},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5893,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5893,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T06:59:18.023 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2024-09-12T06:59:18.023 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-12T06:59:18.301 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:59:18.304 INFO:teuthology.orchestra.run.smithi096.stderr:dumped all 2024-09-12T06:59:18.313 INFO:teuthology.orchestra.run.smithi096.stdout:{"pg_ready":true,"pg_map":{"version":1291,"stamp":"2024-09-12T06:59:16.500287+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4536768,"kb_used_data":8256,"kb_used_omap":462,"kb_used_meta":4527985,"kb_avail":278578752,"statfs":{"total":289910292480,"available":285264642048,"internally_reserved":0,"allocated":8454144,"data_stored":5575555,"data_compressed":11786,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":473703,"internal_metadata":4636657049},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002085"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":678,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-12T06:58:44.978639+0000","last_change":"2024-09-12T06:27:34.239517+0000","last_active":"2024-09-12T06:58:44.978639+0000","last_peered":"2024-09-12T06:58:44.978639+0000","last_clean":"2024-09-12T06:58:44.978639+0000","last_became_active":"2024-09-12T06:27:32.239188+0000","last_became_peered":"2024-09-12T06:27:32.239188+0000","last_unstale":"2024-09-12T06:58:44.978639+0000","last_undegraded":"2024-09-12T06:58:44.978639+0000","last_fullsized":"2024-09-12T06:58:44.978639+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_clean_scrub_stamp":"2024-09-12T06:27:31.224809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T15:03:24.767390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011638639999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":678,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-12T06:58:44.977748+0000","last_change":"2024-09-12T06:27:34.239133+0000","last_active":"2024-09-12T06:58:44.977748+0000","last_peered":"2024-09-12T06:58:44.977748+0000","last_clean":"2024-09-12T06:58:44.977748+0000","last_became_active":"2024-09-12T06:27:32.239159+0000","last_became_peered":"2024-09-12T06:27:32.239159+0000","last_unstale":"2024-09-12T06:58:44.977748+0000","last_undegraded":"2024-09-12T06:58:44.977748+0000","last_fullsized":"2024-09-12T06:58:44.977748+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_clean_scrub_stamp":"2024-09-12T06:27:31.224809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:25:39.727415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079824799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":679,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-12T06:58:44.979807+0000","last_change":"2024-09-12T06:27:34.238743+0000","last_active":"2024-09-12T06:58:44.979807+0000","last_peered":"2024-09-12T06:58:44.979807+0000","last_clean":"2024-09-12T06:58:44.979807+0000","last_became_active":"2024-09-12T06:27:32.239133+0000","last_became_peered":"2024-09-12T06:27:32.239133+0000","last_unstale":"2024-09-12T06:58:44.979807+0000","last_undegraded":"2024-09-12T06:58:44.979807+0000","last_fullsized":"2024-09-12T06:58:44.979807+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_clean_scrub_stamp":"2024-09-12T06:27:31.224809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:11:59.861727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050645400000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":679,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-12T06:58:44.975906+0000","last_change":"2024-09-12T06:27:34.238755+0000","last_active":"2024-09-12T06:58:44.975906+0000","last_peered":"2024-09-12T06:58:44.975906+0000","last_clean":"2024-09-12T06:58:44.975906+0000","last_became_active":"2024-09-12T06:27:32.239390+0000","last_became_peered":"2024-09-12T06:27:32.239390+0000","last_unstale":"2024-09-12T06:58:44.975906+0000","last_undegraded":"2024-09-12T06:58:44.975906+0000","last_fullsized":"2024-09-12T06:58:44.975906+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_clean_scrub_stamp":"2024-09-12T06:27:31.224809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:18:29.532273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050204899999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":680,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-12T06:58:44.979328+0000","last_change":"2024-09-12T06:27:34.242205+0000","last_active":"2024-09-12T06:58:44.979328+0000","last_peered":"2024-09-12T06:58:44.979328+0000","last_clean":"2024-09-12T06:58:44.979328+0000","last_became_active":"2024-09-12T06:27:32.240428+0000","last_became_peered":"2024-09-12T06:27:32.240428+0000","last_unstale":"2024-09-12T06:58:44.979328+0000","last_undegraded":"2024-09-12T06:58:44.979328+0000","last_fullsized":"2024-09-12T06:58:44.979328+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_clean_scrub_stamp":"2024-09-12T06:27:31.224809+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:37:32.701549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095364099999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":679,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-12T06:58:45.695088+0000","last_change":"2024-09-12T06:27:34.508520+0000","last_active":"2024-09-12T06:58:45.695088+0000","last_peered":"2024-09-12T06:58:45.695088+0000","last_clean":"2024-09-12T06:58:45.695088+0000","last_became_active":"2024-09-12T06:27:32.239409+0000","last_became_peered":"2024-09-12T06:27:32.239409+0000","last_unstale":"2024-09-12T06:58:45.695088+0000","last_undegraded":"2024-09-12T06:58:45.695088+0000","last_fullsized":"2024-09-12T06:58:45.695088+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_clean_scrub_stamp":"2024-09-12T06:27:31.224809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T10:21:29.326121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045469599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":679,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-12T06:58:45.694715+0000","last_change":"2024-09-12T06:27:34.508464+0000","last_active":"2024-09-12T06:58:45.694715+0000","last_peered":"2024-09-12T06:58:45.694715+0000","last_clean":"2024-09-12T06:58:45.694715+0000","last_became_active":"2024-09-12T06:27:32.239718+0000","last_became_peered":"2024-09-12T06:27:32.239718+0000","last_unstale":"2024-09-12T06:58:45.694715+0000","last_undegraded":"2024-09-12T06:58:45.694715+0000","last_fullsized":"2024-09-12T06:58:45.694715+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_clean_scrub_stamp":"2024-09-12T06:27:31.224809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:21:30.104428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039987200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":680,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-12T06:58:44.977057+0000","last_change":"2024-09-12T06:27:34.239177+0000","last_active":"2024-09-12T06:58:44.977057+0000","last_peered":"2024-09-12T06:58:44.977057+0000","last_clean":"2024-09-12T06:58:44.977057+0000","last_became_active":"2024-09-12T06:27:32.239878+0000","last_became_peered":"2024-09-12T06:27:32.239878+0000","last_unstale":"2024-09-12T06:58:44.977057+0000","last_undegraded":"2024-09-12T06:58:44.977057+0000","last_fullsized":"2024-09-12T06:58:44.977057+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_clean_scrub_stamp":"2024-09-12T06:27:31.224809+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:11:21.471227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091624399999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'52","reported_seq":747,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-12T06:58:44.976209+0000","last_change":"2024-09-12T06:27:27.078058+0000","last_active":"2024-09-12T06:58:44.976209+0000","last_peered":"2024-09-12T06:58:44.976209+0000","last_clean":"2024-09-12T06:58:44.976209+0000","last_became_active":"2024-09-12T06:27:27.076932+0000","last_became_peered":"2024-09-12T06:27:27.076932+0000","last_unstale":"2024-09-12T06:58:44.976209+0000","last_undegraded":"2024-09-12T06:58:44.976209+0000","last_fullsized":"2024-09-12T06:58:44.976209+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:27:25.120999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:27:25.120999+0000","last_clean_scrub_stamp":"2024-09-12T06:27:25.120999+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:20:08.653460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11786,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":9,"seq":38654706050,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1307752,"kb_used_data":2772,"kb_used_omap":204,"kb_used_meta":1304755,"kb_avail":93064088,"statfs":{"total":96636764160,"available":95297626112,"internally_reserved":0,"allocated":2838528,"data_stored":2055314,"data_compressed":5893,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":209748,"internal_metadata":1336069292},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738754,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1436076,"kb_used_data":2712,"kb_used_omap":135,"kb_used_meta":1433208,"kb_avail":92935764,"statfs":{"total":96636764160,"available":95166222336,"internally_reserved":0,"allocated":2777088,"data_stored":1464927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":139153,"internal_metadata":1467605103},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738754,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1792940,"kb_used_data":2772,"kb_used_omap":121,"kb_used_meta":1790022,"kb_avail":92578900,"statfs":{"total":96636764160,"available":94800793600,"internally_reserved":0,"allocated":2838528,"data_stored":2055314,"data_compressed":5893,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":124802,"internal_metadata":1832982654},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5893,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5893,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T06:59:18.314 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-09-12T06:59:18.599 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:59:18.599 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":337,"fsid":"eb40689d-c0e4-40e2-9b05-3a3a0fa6fbce","created":"2024-09-12T06:27:15.694619+0000","modified":"2024-09-12T06:58:44.967799+0000","last_up_change":"2024-09-12T06:27:25.120999+0000","last_in_change":"2024-09-12T06:27:16.540973+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":3,"require_min_compat_client":"octopus","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-12T06:27:24.257197+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-12T06:27:30.982769+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"eddb0881-a76f-48e4-a872-c9ca4ae8e4ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":3907677134},{"type":"v1","addr":"172.21.15.96:6809","nonce":3907677134}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":3907677134},{"type":"v1","addr":"172.21.15.96:6811","nonce":3907677134}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":3907677134},{"type":"v1","addr":"172.21.15.96:6815","nonce":3907677134}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":3907677134},{"type":"v1","addr":"172.21.15.96:6813","nonce":3907677134}]},"public_addr":"172.21.15.96:6809/3907677134","cluster_addr":"172.21.15.96:6811/3907677134","heartbeat_back_addr":"172.21.15.96:6815/3907677134","heartbeat_front_addr":"172.21.15.96:6813/3907677134","state":["exists","up"]},{"osd":1,"uuid":"e312ae88-6495-4e04-9025-1b8e58a838eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":3125814246},{"type":"v1","addr":"172.21.15.96:6801","nonce":3125814246}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":3125814246},{"type":"v1","addr":"172.21.15.96:6803","nonce":3125814246}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":3125814246},{"type":"v1","addr":"172.21.15.96:6807","nonce":3125814246}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":3125814246},{"type":"v1","addr":"172.21.15.96:6805","nonce":3125814246}]},"public_addr":"172.21.15.96:6801/3125814246","cluster_addr":"172.21.15.96:6803/3125814246","heartbeat_back_addr":"172.21.15.96:6807/3125814246","heartbeat_front_addr":"172.21.15.96:6805/3125814246","state":["exists","up"]},{"osd":2,"uuid":"f2b94172-d7bd-4308-baf0-d55d1f0c2790","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":1528565651},{"type":"v1","addr":"172.21.15.96:6817","nonce":1528565651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":1528565651},{"type":"v1","addr":"172.21.15.96:6819","nonce":1528565651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1528565651},{"type":"v1","addr":"172.21.15.96:6823","nonce":1528565651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":1528565651},{"type":"v1","addr":"172.21.15.96:6821","nonce":1528565651}]},"public_addr":"172.21.15.96:6817/1528565651","cluster_addr":"172.21.15.96:6819/1528565651","heartbeat_back_addr":"172.21.15.96:6823/1528565651","heartbeat_front_addr":"172.21.15.96:6821/1528565651","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T06:27:23.096668+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T06:27:22.767733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T06:27:22.668557+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-12T06:59:19.612 INFO:tasks.ceph:Scrubbing cluster1.osd.0 2024-09-12T06:59:19.613 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-12T06:59:19.747 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-12T06:59:19.747 INFO:teuthology.orchestra.run.smithi096.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-12T06:59:19.747 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-12T06:59:19.757 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 0 2024-09-12T06:59:20.064 INFO:teuthology.orchestra.run.smithi096.stderr:instructed osd(s) 0 to deep-scrub 2024-09-12T06:59:20.077 INFO:tasks.ceph:Scrubbing cluster1.osd.1 2024-09-12T06:59:20.077 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-12T06:59:20.199 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-12T06:59:20.199 INFO:teuthology.orchestra.run.smithi096.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-12T06:59:20.199 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-12T06:59:20.210 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 1 2024-09-12T06:59:20.492 INFO:teuthology.orchestra.run.smithi096.stderr:instructed osd(s) 1 to deep-scrub 2024-09-12T06:59:20.506 INFO:tasks.ceph:Scrubbing cluster1.osd.2 2024-09-12T06:59:20.506 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-12T06:59:20.628 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-12T06:59:20.628 INFO:teuthology.orchestra.run.smithi096.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-12T06:59:20.628 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-12T06:59:20.638 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 2 2024-09-12T06:59:20.925 INFO:teuthology.orchestra.run.smithi096.stderr:instructed osd(s) 2 to deep-scrub 2024-09-12T06:59:20.938 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-12T06:59:21.220 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:59:21.220 INFO:teuthology.orchestra.run.smithi096.stderr:dumped all 2024-09-12T06:59:21.233 INFO:teuthology.orchestra.run.smithi096.stdout:{"pg_ready":true,"pg_map":{"version":1293,"stamp":"2024-09-12T06:59:20.501013+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4536768,"kb_used_data":8256,"kb_used_omap":462,"kb_used_meta":4527985,"kb_avail":278578752,"statfs":{"total":289910292480,"available":285264642048,"internally_reserved":0,"allocated":8454144,"data_stored":5575555,"data_compressed":11786,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":473703,"internal_metadata":4636657049},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002129"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":678,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-12T06:58:44.978639+0000","last_change":"2024-09-12T06:27:34.239517+0000","last_active":"2024-09-12T06:58:44.978639+0000","last_peered":"2024-09-12T06:58:44.978639+0000","last_clean":"2024-09-12T06:58:44.978639+0000","last_became_active":"2024-09-12T06:27:32.239188+0000","last_became_peered":"2024-09-12T06:27:32.239188+0000","last_unstale":"2024-09-12T06:58:44.978639+0000","last_undegraded":"2024-09-12T06:58:44.978639+0000","last_fullsized":"2024-09-12T06:58:44.978639+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_clean_scrub_stamp":"2024-09-12T06:27:31.224809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T15:03:24.767390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011638639999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":678,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-12T06:58:44.977748+0000","last_change":"2024-09-12T06:27:34.239133+0000","last_active":"2024-09-12T06:58:44.977748+0000","last_peered":"2024-09-12T06:58:44.977748+0000","last_clean":"2024-09-12T06:58:44.977748+0000","last_became_active":"2024-09-12T06:27:32.239159+0000","last_became_peered":"2024-09-12T06:27:32.239159+0000","last_unstale":"2024-09-12T06:58:44.977748+0000","last_undegraded":"2024-09-12T06:58:44.977748+0000","last_fullsized":"2024-09-12T06:58:44.977748+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_clean_scrub_stamp":"2024-09-12T06:27:31.224809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:25:39.727415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079824799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":679,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-12T06:58:44.979807+0000","last_change":"2024-09-12T06:27:34.238743+0000","last_active":"2024-09-12T06:58:44.979807+0000","last_peered":"2024-09-12T06:58:44.979807+0000","last_clean":"2024-09-12T06:58:44.979807+0000","last_became_active":"2024-09-12T06:27:32.239133+0000","last_became_peered":"2024-09-12T06:27:32.239133+0000","last_unstale":"2024-09-12T06:58:44.979807+0000","last_undegraded":"2024-09-12T06:58:44.979807+0000","last_fullsized":"2024-09-12T06:58:44.979807+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_clean_scrub_stamp":"2024-09-12T06:27:31.224809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:11:59.861727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050645400000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":679,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-12T06:58:44.975906+0000","last_change":"2024-09-12T06:27:34.238755+0000","last_active":"2024-09-12T06:58:44.975906+0000","last_peered":"2024-09-12T06:58:44.975906+0000","last_clean":"2024-09-12T06:58:44.975906+0000","last_became_active":"2024-09-12T06:27:32.239390+0000","last_became_peered":"2024-09-12T06:27:32.239390+0000","last_unstale":"2024-09-12T06:58:44.975906+0000","last_undegraded":"2024-09-12T06:58:44.975906+0000","last_fullsized":"2024-09-12T06:58:44.975906+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_clean_scrub_stamp":"2024-09-12T06:27:31.224809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:18:29.532273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050204899999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":680,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-12T06:58:44.979328+0000","last_change":"2024-09-12T06:27:34.242205+0000","last_active":"2024-09-12T06:58:44.979328+0000","last_peered":"2024-09-12T06:58:44.979328+0000","last_clean":"2024-09-12T06:58:44.979328+0000","last_became_active":"2024-09-12T06:27:32.240428+0000","last_became_peered":"2024-09-12T06:27:32.240428+0000","last_unstale":"2024-09-12T06:58:44.979328+0000","last_undegraded":"2024-09-12T06:58:44.979328+0000","last_fullsized":"2024-09-12T06:58:44.979328+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_clean_scrub_stamp":"2024-09-12T06:27:31.224809+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:37:32.701549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095364099999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":679,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-12T06:58:45.695088+0000","last_change":"2024-09-12T06:27:34.508520+0000","last_active":"2024-09-12T06:58:45.695088+0000","last_peered":"2024-09-12T06:58:45.695088+0000","last_clean":"2024-09-12T06:58:45.695088+0000","last_became_active":"2024-09-12T06:27:32.239409+0000","last_became_peered":"2024-09-12T06:27:32.239409+0000","last_unstale":"2024-09-12T06:58:45.695088+0000","last_undegraded":"2024-09-12T06:58:45.695088+0000","last_fullsized":"2024-09-12T06:58:45.695088+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_clean_scrub_stamp":"2024-09-12T06:27:31.224809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T10:21:29.326121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045469599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":679,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-12T06:58:45.694715+0000","last_change":"2024-09-12T06:27:34.508464+0000","last_active":"2024-09-12T06:58:45.694715+0000","last_peered":"2024-09-12T06:58:45.694715+0000","last_clean":"2024-09-12T06:58:45.694715+0000","last_became_active":"2024-09-12T06:27:32.239718+0000","last_became_peered":"2024-09-12T06:27:32.239718+0000","last_unstale":"2024-09-12T06:58:45.694715+0000","last_undegraded":"2024-09-12T06:58:45.694715+0000","last_fullsized":"2024-09-12T06:58:45.694715+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_clean_scrub_stamp":"2024-09-12T06:27:31.224809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:21:30.104428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039987200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":680,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-12T06:58:44.977057+0000","last_change":"2024-09-12T06:27:34.239177+0000","last_active":"2024-09-12T06:58:44.977057+0000","last_peered":"2024-09-12T06:58:44.977057+0000","last_clean":"2024-09-12T06:58:44.977057+0000","last_became_active":"2024-09-12T06:27:32.239878+0000","last_became_peered":"2024-09-12T06:27:32.239878+0000","last_unstale":"2024-09-12T06:58:44.977057+0000","last_undegraded":"2024-09-12T06:58:44.977057+0000","last_fullsized":"2024-09-12T06:58:44.977057+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:27:31.224809+0000","last_clean_scrub_stamp":"2024-09-12T06:27:31.224809+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:11:21.471227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091624399999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'52","reported_seq":747,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-12T06:58:44.976209+0000","last_change":"2024-09-12T06:27:27.078058+0000","last_active":"2024-09-12T06:58:44.976209+0000","last_peered":"2024-09-12T06:58:44.976209+0000","last_clean":"2024-09-12T06:58:44.976209+0000","last_became_active":"2024-09-12T06:27:27.076932+0000","last_became_peered":"2024-09-12T06:27:27.076932+0000","last_unstale":"2024-09-12T06:58:44.976209+0000","last_undegraded":"2024-09-12T06:58:44.976209+0000","last_fullsized":"2024-09-12T06:58:44.976209+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:27:25.120999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:27:25.120999+0000","last_clean_scrub_stamp":"2024-09-12T06:27:25.120999+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:20:08.653460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11786,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":9,"seq":38654706050,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1307752,"kb_used_data":2772,"kb_used_omap":204,"kb_used_meta":1304755,"kb_avail":93064088,"statfs":{"total":96636764160,"available":95297626112,"internally_reserved":0,"allocated":2838528,"data_stored":2055314,"data_compressed":5893,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":209748,"internal_metadata":1336069292},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738755,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1436076,"kb_used_data":2712,"kb_used_omap":135,"kb_used_meta":1433208,"kb_avail":92935764,"statfs":{"total":96636764160,"available":95166222336,"internally_reserved":0,"allocated":2777088,"data_stored":1464927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":139153,"internal_metadata":1467605103},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738755,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1792940,"kb_used_data":2772,"kb_used_omap":121,"kb_used_meta":1790022,"kb_avail":92578900,"statfs":{"total":96636764160,"available":94800793600,"internally_reserved":0,"allocated":2838528,"data_stored":2055314,"data_compressed":5893,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":124802,"internal_metadata":1832982654},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5893,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5893,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T06:59:21.234 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-12T06:27:31.224809+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=6, tm_min=27, tm_sec=31, tm_wday=3, tm_yday=256, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=6, tm_min=59, tm_sec=18, tm_wday=3, tm_yday=256, tm_isdst=0) 2024-09-12T06:59:21.234 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-12T06:27:31.224809+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=6, tm_min=27, tm_sec=31, tm_wday=3, tm_yday=256, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=6, tm_min=59, tm_sec=18, tm_wday=3, tm_yday=256, tm_isdst=0) 2024-09-12T06:59:21.234 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-12T06:27:31.224809+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=6, tm_min=27, tm_sec=31, tm_wday=3, tm_yday=256, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=6, tm_min=59, tm_sec=18, tm_wday=3, tm_yday=256, tm_isdst=0) 2024-09-12T06:59:21.234 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-12T06:27:31.224809+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=6, tm_min=27, tm_sec=31, tm_wday=3, tm_yday=256, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=6, tm_min=59, tm_sec=18, tm_wday=3, tm_yday=256, tm_isdst=0) 2024-09-12T06:59:21.234 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-12T06:27:31.224809+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=6, tm_min=27, tm_sec=31, tm_wday=3, tm_yday=256, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=6, tm_min=59, tm_sec=18, tm_wday=3, tm_yday=256, tm_isdst=0) 2024-09-12T06:59:21.234 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-12T06:27:31.224809+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=6, tm_min=27, tm_sec=31, tm_wday=3, tm_yday=256, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=6, tm_min=59, tm_sec=18, tm_wday=3, tm_yday=256, tm_isdst=0) 2024-09-12T06:59:21.234 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-12T06:27:31.224809+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=6, tm_min=27, tm_sec=31, tm_wday=3, tm_yday=256, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=6, tm_min=59, tm_sec=18, tm_wday=3, tm_yday=256, tm_isdst=0) 2024-09-12T06:59:21.235 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-12T06:27:31.224809+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=6, tm_min=27, tm_sec=31, tm_wday=3, tm_yday=256, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=6, tm_min=59, tm_sec=18, tm_wday=3, tm_yday=256, tm_isdst=0) 2024-09-12T06:59:21.235 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-12T06:27:25.120999+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=6, tm_min=27, tm_sec=25, tm_wday=3, tm_yday=256, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=6, tm_min=59, tm_sec=18, tm_wday=3, tm_yday=256, tm_isdst=0) 2024-09-12T06:59:21.235 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-12T06:59:41.236 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-12T06:59:41.531 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-12T06:59:41.532 INFO:teuthology.orchestra.run.smithi096.stderr:dumped all 2024-09-12T06:59:41.544 INFO:teuthology.orchestra.run.smithi096.stdout:{"pg_ready":true,"pg_map":{"version":1303,"stamp":"2024-09-12T06:59:40.504755+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4536752,"kb_used_data":8256,"kb_used_omap":462,"kb_used_meta":4527985,"kb_avail":278578768,"statfs":{"total":289910292480,"available":285264658432,"internally_reserved":0,"allocated":8454144,"data_stored":5575555,"data_compressed":11786,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":473703,"internal_metadata":4636657049},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002362"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":686,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-12T06:59:24.747431+0000","last_change":"2024-09-12T06:59:24.747431+0000","last_active":"2024-09-12T06:59:24.747431+0000","last_peered":"2024-09-12T06:59:24.747431+0000","last_clean":"2024-09-12T06:59:24.747431+0000","last_became_active":"2024-09-12T06:27:32.239188+0000","last_became_peered":"2024-09-12T06:27:32.239188+0000","last_unstale":"2024-09-12T06:59:24.747431+0000","last_undegraded":"2024-09-12T06:59:24.747431+0000","last_fullsized":"2024-09-12T06:59:24.747431+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:59:24.747344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:59:24.747344+0000","last_clean_scrub_stamp":"2024-09-12T06:59:24.747344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:54:56.805565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011638639999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":686,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-12T06:59:25.756333+0000","last_change":"2024-09-12T06:59:25.756333+0000","last_active":"2024-09-12T06:59:25.756333+0000","last_peered":"2024-09-12T06:59:25.756333+0000","last_clean":"2024-09-12T06:59:25.756333+0000","last_became_active":"2024-09-12T06:27:32.239159+0000","last_became_peered":"2024-09-12T06:27:32.239159+0000","last_unstale":"2024-09-12T06:59:25.756333+0000","last_undegraded":"2024-09-12T06:59:25.756333+0000","last_fullsized":"2024-09-12T06:59:25.756333+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:59:25.756244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:59:25.756244+0000","last_clean_scrub_stamp":"2024-09-12T06:59:25.756244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:25:27.181239+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00079824799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":687,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-12T06:59:22.772359+0000","last_change":"2024-09-12T06:59:22.772359+0000","last_active":"2024-09-12T06:59:22.772359+0000","last_peered":"2024-09-12T06:59:22.772359+0000","last_clean":"2024-09-12T06:59:22.772359+0000","last_became_active":"2024-09-12T06:27:32.239133+0000","last_became_peered":"2024-09-12T06:27:32.239133+0000","last_unstale":"2024-09-12T06:59:22.772359+0000","last_undegraded":"2024-09-12T06:59:22.772359+0000","last_fullsized":"2024-09-12T06:59:22.772359+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:59:22.772273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:59:22.772273+0000","last_clean_scrub_stamp":"2024-09-12T06:59:22.772273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:44:09.586841+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00050645400000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":687,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-12T06:59:23.765564+0000","last_change":"2024-09-12T06:59:23.765564+0000","last_active":"2024-09-12T06:59:23.765564+0000","last_peered":"2024-09-12T06:59:23.765564+0000","last_clean":"2024-09-12T06:59:23.765564+0000","last_became_active":"2024-09-12T06:27:32.239390+0000","last_became_peered":"2024-09-12T06:27:32.239390+0000","last_unstale":"2024-09-12T06:59:23.765564+0000","last_undegraded":"2024-09-12T06:59:23.765564+0000","last_fullsized":"2024-09-12T06:59:23.765564+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:59:23.765460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:59:23.765460+0000","last_clean_scrub_stamp":"2024-09-12T06:59:23.765460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T07:40:52.771172+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00050204899999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"337'3","reported_seq":689,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-12T06:59:20.862705+0000","last_change":"2024-09-12T06:59:20.862705+0000","last_active":"2024-09-12T06:59:20.862705+0000","last_peered":"2024-09-12T06:59:20.862705+0000","last_clean":"2024-09-12T06:59:20.862705+0000","last_became_active":"2024-09-12T06:27:32.240428+0000","last_became_peered":"2024-09-12T06:27:32.240428+0000","last_unstale":"2024-09-12T06:59:20.862705+0000","last_undegraded":"2024-09-12T06:59:20.862705+0000","last_fullsized":"2024-09-12T06:59:20.862705+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"337'3","last_scrub_stamp":"2024-09-12T06:59:20.862618+0000","last_deep_scrub":"337'3","last_deep_scrub_stamp":"2024-09-12T06:59:20.862618+0000","last_clean_scrub_stamp":"2024-09-12T06:59:20.862618+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T14:44:51.889386+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00095364099999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":687,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-12T06:59:21.763832+0000","last_change":"2024-09-12T06:59:21.763832+0000","last_active":"2024-09-12T06:59:21.763832+0000","last_peered":"2024-09-12T06:59:21.763832+0000","last_clean":"2024-09-12T06:59:21.763832+0000","last_became_active":"2024-09-12T06:27:32.239409+0000","last_became_peered":"2024-09-12T06:27:32.239409+0000","last_unstale":"2024-09-12T06:59:21.763832+0000","last_undegraded":"2024-09-12T06:59:21.763832+0000","last_fullsized":"2024-09-12T06:59:21.763832+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:59:21.763731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:59:21.763731+0000","last_clean_scrub_stamp":"2024-09-12T06:59:21.763731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T16:44:27.431108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045469599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":687,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-12T06:59:22.755517+0000","last_change":"2024-09-12T06:59:22.755517+0000","last_active":"2024-09-12T06:59:22.755517+0000","last_peered":"2024-09-12T06:59:22.755517+0000","last_clean":"2024-09-12T06:59:22.755517+0000","last_became_active":"2024-09-12T06:27:32.239718+0000","last_became_peered":"2024-09-12T06:27:32.239718+0000","last_unstale":"2024-09-12T06:59:22.755517+0000","last_undegraded":"2024-09-12T06:59:22.755517+0000","last_fullsized":"2024-09-12T06:59:22.755517+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T06:59:22.755432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T06:59:22.755432+0000","last_clean_scrub_stamp":"2024-09-12T06:59:22.755432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T15:07:32.585150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039987200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"337'2","reported_seq":689,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-12T06:59:21.773321+0000","last_change":"2024-09-12T06:59:21.773321+0000","last_active":"2024-09-12T06:59:21.773321+0000","last_peered":"2024-09-12T06:59:21.773321+0000","last_clean":"2024-09-12T06:59:21.773321+0000","last_became_active":"2024-09-12T06:27:32.239878+0000","last_became_peered":"2024-09-12T06:27:32.239878+0000","last_unstale":"2024-09-12T06:59:21.773321+0000","last_undegraded":"2024-09-12T06:59:21.773321+0000","last_fullsized":"2024-09-12T06:59:21.773321+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"337'2","last_scrub_stamp":"2024-09-12T06:59:21.773253+0000","last_deep_scrub":"337'2","last_deep_scrub_stamp":"2024-09-12T06:59:21.773253+0000","last_clean_scrub_stamp":"2024-09-12T06:59:21.773253+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:23:08.754270+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00091624399999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"337'54","reported_seq":758,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-12T06:59:20.758838+0000","last_change":"2024-09-12T06:59:20.758762+0000","last_active":"2024-09-12T06:59:20.758838+0000","last_peered":"2024-09-12T06:59:20.758838+0000","last_clean":"2024-09-12T06:59:20.758838+0000","last_became_active":"2024-09-12T06:27:27.076932+0000","last_became_peered":"2024-09-12T06:27:27.076932+0000","last_unstale":"2024-09-12T06:59:20.758838+0000","last_undegraded":"2024-09-12T06:59:20.758838+0000","last_fullsized":"2024-09-12T06:59:20.758838+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"337'54","last_scrub_stamp":"2024-09-12T06:59:20.758657+0000","last_deep_scrub":"337'54","last_deep_scrub_stamp":"2024-09-12T06:59:20.758657+0000","last_clean_scrub_stamp":"2024-09-12T06:59:20.758657+0000","objects_scrubbed":2,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T17:25:27.305324+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11786,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":54,"ondisk_log_size":54,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":9,"seq":38654706054,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1307748,"kb_used_data":2772,"kb_used_omap":204,"kb_used_meta":1304755,"kb_avail":93064092,"statfs":{"total":96636764160,"available":95297630208,"internally_reserved":0,"allocated":2838528,"data_stored":2055314,"data_compressed":5893,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":209748,"internal_metadata":1336069292},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738759,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1436072,"kb_used_data":2712,"kb_used_omap":135,"kb_used_meta":1433208,"kb_avail":92935768,"statfs":{"total":96636764160,"available":95166226432,"internally_reserved":0,"allocated":2777088,"data_stored":1464927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":139153,"internal_metadata":1467605103},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738759,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1792932,"kb_used_data":2772,"kb_used_omap":121,"kb_used_meta":1790022,"kb_avail":92578908,"statfs":{"total":96636764160,"available":94800801792,"internally_reserved":0,"allocated":2838528,"data_stored":2055314,"data_compressed":5893,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":124802,"internal_metadata":1832982654},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5893,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5893,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T06:59:41.546 DEBUG:teuthology.orchestra.run.smithi096:> sudo ceph --cluster cluster1 config set global mon_health_to_clog false 2024-09-12T06:59:41.843 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-12T06:59:41.843 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-12T06:59:41.844 DEBUG:tasks.ceph.cluster1.osd.0:waiting for process to exit 2024-09-12T06:59:41.844 INFO:teuthology.orchestra.run:waiting for 300 2024-09-12T06:59:41.914 INFO:tasks.ceph.cluster1.osd.0:Stopped 2024-09-12T06:59:41.914 DEBUG:tasks.ceph.cluster1.osd.1:waiting for process to exit 2024-09-12T06:59:41.914 INFO:teuthology.orchestra.run:waiting for 300 2024-09-12T06:59:42.001 INFO:tasks.ceph.cluster1.osd.1:Stopped 2024-09-12T06:59:42.001 DEBUG:tasks.ceph.cluster1.osd.2:waiting for process to exit 2024-09-12T06:59:42.002 INFO:teuthology.orchestra.run:waiting for 300 2024-09-12T06:59:42.072 INFO:tasks.ceph.cluster1.osd.2:Stopped 2024-09-12T06:59:42.072 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-12T06:59:42.072 DEBUG:tasks.ceph.cluster1.mgr.x:waiting for process to exit 2024-09-12T06:59:42.073 INFO:teuthology.orchestra.run:waiting for 300 2024-09-12T06:59:42.099 INFO:tasks.ceph.cluster1.mgr.x:Stopped 2024-09-12T06:59:42.099 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-12T06:59:42.099 DEBUG:tasks.ceph.cluster1.mon.a:waiting for process to exit 2024-09-12T06:59:42.100 INFO:teuthology.orchestra.run:waiting for 300 2024-09-12T06:59:42.152 INFO:tasks.ceph.cluster1.mon.a:Stopped 2024-09-12T06:59:42.153 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-12T06:59:42.153 DEBUG:teuthology.orchestra.run.smithi096:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-09-12T06:59:42.256 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-0 on ubuntu@smithi096.front.sepia.ceph.com 2024-09-12T06:59:42.257 DEBUG:teuthology.orchestra.run.smithi096:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-0 2024-09-12T06:59:42.679 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-1 on ubuntu@smithi096.front.sepia.ceph.com 2024-09-12T06:59:42.679 DEBUG:teuthology.orchestra.run.smithi096:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-1 2024-09-12T06:59:42.919 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-2 on ubuntu@smithi096.front.sepia.ceph.com 2024-09-12T06:59:42.919 DEBUG:teuthology.orchestra.run.smithi096:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-2 2024-09-12T06:59:43.182 INFO:tasks.ceph:Archiving mon data... 2024-09-12T06:59:43.182 DEBUG:teuthology.misc:Transferring archived files from smithi096:/var/lib/ceph/mon/cluster1-a to /home/teuthworker/archive/dis-2024-09-12_04:46:10-rbd-wip-dis-testing-distro-default-smithi/7902364/data/cluster1.mon.a.tgz 2024-09-12T06:59:43.183 DEBUG:teuthology.orchestra.run.smithi096:> mktemp 2024-09-12T06:59:43.207 INFO:teuthology.orchestra.run.smithi096.stdout:/tmp/tmp.LPUUuLZ5PG 2024-09-12T06:59:43.208 DEBUG:teuthology.orchestra.run.smithi096:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster1-a -- . > /tmp/tmp.LPUUuLZ5PG 2024-09-12T06:59:43.402 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod 0666 /tmp/tmp.LPUUuLZ5PG 2024-09-12T06:59:43.500 DEBUG:teuthology.orchestra.remote:smithi096:/tmp/tmp.LPUUuLZ5PG is 397KB 2024-09-12T06:59:43.598 DEBUG:teuthology.orchestra.run.smithi096:> rm -fr /tmp/tmp.LPUUuLZ5PG 2024-09-12T06:59:43.613 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-12T06:59:43.613 DEBUG:teuthology.orchestra.run.smithi096:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2024-09-12T06:59:43.655 DEBUG:teuthology.orchestra.run.smithi152:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2024-09-12T06:59:43.746 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-12T06:59:43.765 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-12T06:59:43.765 INFO:tasks.ceph:Archiving crash dumps... 2024-09-12T06:59:43.767 DEBUG:teuthology.misc:Transferring archived files from smithi096:/var/lib/ceph/crash to /home/teuthworker/archive/dis-2024-09-12_04:46:10-rbd-wip-dis-testing-distro-default-smithi/7902364/remote/smithi096/crash 2024-09-12T06:59:43.768 DEBUG:teuthology.orchestra.run.smithi096:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-12T06:59:43.811 DEBUG:teuthology.misc:Transferring archived files from smithi152:/var/lib/ceph/crash to /home/teuthworker/archive/dis-2024-09-12_04:46:10-rbd-wip-dis-testing-distro-default-smithi/7902364/remote/smithi152/crash 2024-09-12T06:59:43.812 DEBUG:teuthology.orchestra.run.smithi152:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-12T06:59:43.840 INFO:tasks.ceph:Compressing logs... 2024-09-12T06:59:43.840 DEBUG:teuthology.orchestra.run.smithi096:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-12T06:59:43.856 DEBUG:teuthology.orchestra.run.smithi152:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-12T06:59:43.909 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32165.log 2024-09-12T06:59:43.909 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36908.log 2024-09-12T06:59:43.909 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.32165.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37158.log 2024-09-12T06:59:43.909 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.32165.log.gz 2024-09-12T06:59:43.909 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.36908.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35655.log 2024-09-12T06:59:43.909 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36908.log.gz 2024-09-12T06:59:43.909 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.37158.log: gzip -5 --verbose -- /var/log/ceph/cluster1-mon.a.log 2024-09-12T06:59:43.910 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37158.log.gz 2024-09-12T06:59:43.910 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.35655.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.31967.log 2024-09-12T06:59:43.910 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35655.log.gz 2024-09-12T06:59:43.910 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-mon.a.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32374.log 2024-09-12T06:59:43.910 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.31967.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35445.log 2024-09-12T06:59:43.910 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.31967.log.gz 2024-09-12T06:59:43.910 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.32374.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36992.log 2024-09-12T06:59:43.910 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.35445.log: gzip -5 --verbose -- /var/log/ceph/cluster1-osd.1.log 2024-09-12T06:59:43.910 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35445.log.gz 2024-09-12T06:59:43.910 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.36992.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36656.log 2024-09-12T06:59:43.911 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36992.log.gz 2024-09-12T06:59:43.911 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-osd.1.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35285.log 2024-09-12T06:59:43.911 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.36656.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37076.log 2024-09-12T06:59:43.911 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36656.log.gz 2024-09-12T06:59:43.911 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.35285.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32205.log 2024-09-12T06:59:43.911 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35285.log.gz 2024-09-12T06:59:43.911 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.37076.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35949.log 2024-09-12T06:59:43.911 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37076.log.gz 2024-09-12T06:59:43.911 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.32205.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35310.log 2024-09-12T06:59:43.912 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.32205.log.gz 2024-09-12T06:59:43.912 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.35949.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35991.log 2024-09-12T06:59:43.912 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35949.log.gz 2024-09-12T06:59:43.912 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.35310.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37118.log 2024-09-12T06:59:43.912 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.35991.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35567.log 2024-09-12T06:59:43.912 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35991.log.gz 2024-09-12T06:59:43.912 INFO:teuthology.orchestra.run.smithi096.stderr: 80.1% -- replaced with /var/log/ceph/cluster1-client.admin.35310.log.gz/var/log/ceph/cluster1-client.admin.37118.log: 2024-09-12T06:59:43.912 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36740.log 2024-09-12T06:59:43.912 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37118.log.gz 2024-09-12T06:59:43.913 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.35567.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32304.log 2024-09-12T06:59:43.913 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35567.log.gz 2024-09-12T06:59:43.913 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.36740.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35694.log 2024-09-12T06:59:43.913 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36740.log.gz 2024-09-12T06:59:43.913 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.32304.log: gzip -5 --verbose -- /var/log/ceph/cluster1-osd.2.log 2024-09-12T06:59:43.913 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.32304.log.gz 2024-09-12T06:59:43.913 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.35694.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35107.log 2024-09-12T06:59:43.913 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35694.log.gz 2024-09-12T06:59:43.913 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-osd.2.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36866.log 2024-09-12T06:59:43.914 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.35107.log: gzip -5 --verbose -- /var/log/ceph/cluster2-mgr.x.log 2024-09-12T06:59:43.914 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35107.log.gz 2024-09-12T06:59:43.914 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.36866.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35487.log 2024-09-12T06:59:43.914 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36866.log.gz 2024-09-12T06:59:43.914 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster2-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35569.log 2024-09-12T06:59:43.914 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.35487.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35229.log 2024-09-12T06:59:43.914 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35487.log.gz 2024-09-12T06:59:43.914 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.35569.log: gzip -5 --verbose -- /var/log/ceph/cluster1-mgr.x.log 2024-09-12T06:59:43.914 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35569.log.gz 2024-09-12T06:59:43.914 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.35229.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36698.log 2024-09-12T06:59:43.915 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35229.log.gz 2024-09-12T06:59:43.915 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36950.log 2024-09-12T06:59:43.915 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.36698.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35568.log 2024-09-12T06:59:43.915 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36698.log.gz 2024-09-12T06:59:43.915 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.36950.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35907.log 2024-09-12T06:59:43.915 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36950.log.gz 2024-09-12T06:59:43.915 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.35568.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35820.log 2024-09-12T06:59:43.915 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35568.log.gz 2024-09-12T06:59:43.915 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.35907.log: gzip -5 --verbose -- /var/log/ceph/cluster1.audit.log 2024-09-12T06:59:43.915 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35907.log.gz 2024-09-12T06:59:43.915 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.35820.log: gzip -5 --verbose -- /var/log/ceph/cluster1-osd.0.log 2024-09-12T06:59:43.916 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35820.log.gz 2024-09-12T06:59:43.916 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1.audit.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35695.log 2024-09-12T06:59:43.916 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.27488.log 2024-09-12T06:59:43.916 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.35695.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36782.log 2024-09-12T06:59:43.916 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35695.log.gz 2024-09-12T06:59:43.916 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph.tmp-client.admin.27488.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32125.log 2024-09-12T06:59:43.916 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.27488.log.gz 2024-09-12T06:59:43.916 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.36782.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36824.log 2024-09-12T06:59:43.916 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36782.log.gz 2024-09-12T06:59:43.916 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.32125.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35361.log 2024-09-12T06:59:43.916 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.32125.log.gz 2024-09-12T06:59:43.917 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.36824.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35781.log 2024-09-12T06:59:43.917 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36824.log.gz 2024-09-12T06:59:43.917 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.35361.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35821.log 2024-09-12T06:59:43.917 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35361.log.gz 2024-09-12T06:59:43.917 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.35781.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37034.log 2024-09-12T06:59:43.917 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35781.log.gz 2024-09-12T06:59:43.917 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.35821.log: gzip -5 --verbose -- /var/log/ceph/cluster1.log 2024-09-12T06:59:43.917 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35821.log.gz 2024-09-12T06:59:43.917 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.37034.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35403.log 2024-09-12T06:59:43.917 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37034.log.gz 2024-09-12T06:59:43.917 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1.log: /var/log/ceph/cluster1-client.admin.35403.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35403.log.gz 2024-09-12T06:59:43.917 INFO:teuthology.orchestra.run.smithi096.stderr: 87.0% -- replaced with /var/log/ceph/cluster1.log.gz 2024-09-12T06:59:43.934 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61850.log 2024-09-12T06:59:43.934 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62196.log 2024-09-12T06:59:43.934 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43511.log 2024-09-12T06:59:43.934 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.61850.log: /var/log/ceph/cluster2-client.mirror.62196.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82357.log 2024-09-12T06:59:43.934 INFO:teuthology.orchestra.run.smithi152.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.61850.log.gz 2024-09-12T06:59:43.934 INFO:teuthology.orchestra.run.smithi152.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.62196.log.gz 2024-09-12T06:59:43.934 INFO:teuthology.orchestra.run.smithi152.stderr:gzip/var/log/ceph/cluster2-client.mirror.43511.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71612.log 2024-09-12T06:59:43.935 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.82357.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43464.log 2024-09-12T06:59:43.935 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.71612.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83160.log 2024-09-12T06:59:43.935 INFO:teuthology.orchestra.run.smithi152.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.71612.log.gz 2024-09-12T06:59:43.935 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.43464.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.82357.log.gz 2024-09-12T06:59:43.935 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72508.log 2024-09-12T06:59:43.935 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.83160.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48976.log 2024-09-12T06:59:43.935 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.72508.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82898.log 2024-09-12T06:59:43.935 INFO:teuthology.orchestra.run.smithi152.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.72508.log.gz 2024-09-12T06:59:43.935 INFO:teuthology.orchestra.run.smithi152.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.83160.log.gz 2024-09-12T06:59:43.935 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.48976.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69632.log 2024-09-12T06:59:43.935 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.82898.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83089.log 2024-09-12T06:59:43.935 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.69632.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.82898.log.gz 2024-09-12T06:59:43.935 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42065.log 2024-09-12T06:59:43.936 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.83089.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80683.log 2024-09-12T06:59:43.936 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.42065.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36548.log 2024-09-12T06:59:43.936 INFO:teuthology.orchestra.run.smithi152.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.83089.log.gz 2024-09-12T06:59:43.936 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86936.log 2024-09-12T06:59:43.936 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.80683.log: /var/log/ceph/cluster1-client.admin.36548.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88057.log 2024-09-12T06:59:43.936 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36548.log.gz 2024-09-12T06:59:43.936 INFO:teuthology.orchestra.run.smithi152.stderr: 91.8% -- replaced with /var/log/ceph/cluster1-client.mirror.69632.log.gz 2024-09-12T06:59:43.936 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.86936.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76173.log 2024-09-12T06:59:43.936 INFO:teuthology.orchestra.run.smithi152.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.80683.log.gz 2024-09-12T06:59:43.936 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.88057.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.86936.log.gz 2024-09-12T06:59:43.937 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89402.log 2024-09-12T06:59:43.937 INFO:teuthology.orchestra.run.smithi152.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.88057.log.gz 2024-09-12T06:59:43.937 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.76173.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70982.log 2024-09-12T06:59:43.937 INFO:teuthology.orchestra.run.smithi152.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.76173.log.gz 2024-09-12T06:59:43.937 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.89402.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67362.log 2024-09-12T06:59:43.937 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.70982.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80825.log 2024-09-12T06:59:43.937 INFO:teuthology.orchestra.run.smithi152.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.89402.log.gz 2024-09-12T06:59:43.937 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.67362.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79751.log 2024-09-12T06:59:43.937 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.80825.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44547.log 2024-09-12T06:59:43.938 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.79751.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86707.log 2024-09-12T06:59:43.938 INFO:teuthology.orchestra.run.smithi152.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.80825.log.gz/var/log/ceph/cluster2-client.mirror.44547.log: 2024-09-12T06:59:43.938 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41412.log 2024-09-12T06:59:43.938 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41447.log 2024-09-12T06:59:43.938 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.86707.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.79751.log.gz 2024-09-12T06:59:43.938 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.41412.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77902.log 2024-09-12T06:59:43.938 INFO:teuthology.orchestra.run.smithi152.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.86707.log.gz 2024-09-12T06:59:43.939 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62614.log 2024-09-12T06:59:43.939 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.41447.log: /var/log/ceph/cluster1-client.mirror.77902.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77670.log 2024-09-12T06:59:43.939 INFO:teuthology.orchestra.run.smithi152.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.77902.log.gz 2024-09-12T06:59:43.939 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.62614.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86614.log 2024-09-12T06:59:43.939 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45902.log 2024-09-12T06:59:43.939 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.77670.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.62614.log.gz 2024-09-12T06:59:43.939 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.86614.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85445.log 2024-09-12T06:59:43.939 INFO:teuthology.orchestra.run.smithi152.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.77670.log.gz 2024-09-12T06:59:43.939 INFO:teuthology.orchestra.run.smithi152.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.86614.log.gz 2024-09-12T06:59:43.940 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.45902.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80563.log 2024-09-12T06:59:43.940 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69654.log 2024-09-12T06:59:43.940 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.85445.log: /var/log/ceph/cluster2-client.mirror.80563.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.85445.log.gz 2024-09-12T06:59:43.940 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47397.log 2024-09-12T06:59:43.940 INFO:teuthology.orchestra.run.smithi152.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.80563.log.gz 2024-09-12T06:59:43.940 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.69654.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44371.log 2024-09-12T06:59:43.940 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53101.log 2024-09-12T06:59:43.940 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.47397.log: /var/log/ceph/cluster2-client.mirror.44371.log: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47397.log.gz 2024-09-12T06:59:43.940 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61489.log 2024-09-12T06:59:43.940 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68756.log 2024-09-12T06:59:43.940 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.53101.log: /var/log/ceph/cluster2-client.mirror.61489.log: 84.3% -- replaced with /var/log/ceph/cluster2-client.mirror.53101.log.gz 2024-09-12T06:59:43.941 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78697.log 2024-09-12T06:59:43.941 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.68756.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48735.log 2024-09-12T06:59:43.941 INFO:teuthology.orchestra.run.smithi152.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.68756.log.gz 2024-09-12T06:59:43.941 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.78697.log: gzip -5 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.78697.log.gz 2024-09-12T06:59:43.941 INFO:teuthology.orchestra.run.smithi152.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.50980.log 2024-09-12T06:59:43.941 INFO:teuthology.orchestra.run.smithi152.stderr: 91.9% -- replaced with /var/log/ceph/cluster2-client.mirror.61489.log.gz 2024-09-12T06:59:43.941 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.48735.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60500.log 2024-09-12T06:59:43.941 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.50980.log: 84.4% -- replaced with /var/log/ceph/cluster2-client.mirror.50980.log.gz 2024-09-12T06:59:43.941 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.60500.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87829.log 2024-09-12T06:59:43.941 INFO:teuthology.orchestra.run.smithi152.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.60500.log.gz 2024-09-12T06:59:43.941 INFO:teuthology.orchestra.run.smithi152.stderr: 95.4% -- replaced with /var/log/ceph/cluster2-client.mirror.70982.log.gz 2024-09-12T06:59:43.944 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36083.log 2024-09-12T06:59:43.945 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.87829.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60147.log 2024-09-12T06:59:43.945 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.admin.36083.log: gzip 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36083.log.gz 2024-09-12T06:59:43.945 INFO:teuthology.orchestra.run.smithi152.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80156.log 2024-09-12T06:59:43.945 INFO:teuthology.orchestra.run.smithi152.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.87829.log.gz 2024-09-12T06:59:43.945 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.60147.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39468.log 2024-09-12T06:59:43.945 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.80156.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86914.log 2024-09-12T06:59:43.945 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.39468.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.80156.log.gz 2024-09-12T06:59:43.945 INFO:teuthology.orchestra.run.smithi152.stderr: 91.9% -- replaced with /var/log/ceph/cluster2-client.mirror.60147.log.gz 2024-09-12T06:59:43.945 INFO:teuthology.orchestra.run.smithi152.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.39468.log.gz 2024-09-12T06:59:43.945 INFO:teuthology.orchestra.run.smithi152.stderr: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.45902.log.gz 2024-09-12T06:59:43.947 INFO:teuthology.orchestra.run.smithi152.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.41412.log.gz 2024-09-12T06:59:43.947 INFO:teuthology.orchestra.run.smithi152.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.41447.log.gz 2024-09-12T06:59:43.948 INFO:teuthology.orchestra.run.smithi152.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.42065.log.gz 2024-09-12T06:59:43.954 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71269.log 2024-09-12T06:59:43.980 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.86914.log: gzip 92.5% -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.38047.log 2024-09-12T06:59:43.980 INFO:teuthology.orchestra.run.smithi152.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.44547.log.gz 2024-09-12T06:59:43.981 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.71269.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39436.log 2024-09-12T06:59:43.981 INFO:teuthology.orchestra.run.smithi152.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.86914.log.gz 2024-09-12T06:59:43.981 INFO:teuthology.orchestra.run.smithi152.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.71269.log.gz 2024-09-12T06:59:43.981 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84950.log 2024-09-12T06:59:43.981 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.1.38047.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65695.log 2024-09-12T06:59:43.981 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.39436.log: /var/log/ceph/cluster1-client.mirror.84950.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40780.log 2024-09-12T06:59:43.981 INFO:teuthology.orchestra.run.smithi152.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.84950.log.gz 2024-09-12T06:59:43.981 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.65695.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84221.log 2024-09-12T06:59:43.981 INFO:teuthology.orchestra.run.smithi152.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.39436.log.gz 2024-09-12T06:59:43.981 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.admin.40780.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40968.log 2024-09-12T06:59:43.981 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5/var/log/ceph/cluster2-client.mirror.84221.log: --verbose -- /var/log/ceph/cluster2-client.mirror.66167.log 2024-09-12T06:59:43.981 INFO:teuthology.orchestra.run.smithi152.stderr: 88.4%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71301.log 2024-09-12T06:59:43.981 INFO:teuthology.orchestra.run.smithi152.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.40780.log.gz 2024-09-12T06:59:43.981 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.66167.log: /var/log/ceph/cluster2-client.mirror.40968.log: 91.9% -- replaced with /var/log/ceph/cluster2-client.mirror.65695.log.gz 2024-09-12T06:59:43.982 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45925.log 2024-09-12T06:59:43.982 INFO:teuthology.orchestra.run.smithi152.stderr: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.84221.log.gz 2024-09-12T06:59:43.982 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.71301.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.66167.log.gz 2024-09-12T06:59:43.982 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84477.log 2024-09-12T06:59:43.982 INFO:teuthology.orchestra.run.smithi152.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.71301.log.gz 2024-09-12T06:59:43.982 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.45925.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88776.log 2024-09-12T06:59:43.982 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.84477.log: gzip -5 --verbose -- 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.84477.log.gz 2024-09-12T06:59:43.982 INFO:teuthology.orchestra.run.smithi152.stderr: /var/log/ceph/cluster2-client.mirror.40331.log 2024-09-12T06:59:43.982 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.88776.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73161.log 2024-09-12T06:59:43.982 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.40331.log: gzip -5 --verbose 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.43511.log.gz 2024-09-12T06:59:43.982 INFO:teuthology.orchestra.run.smithi152.stderr: -- /var/log/ceph/cluster2-client.mirror.70848.log 2024-09-12T06:59:43.982 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.73161.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86390.log 2024-09-12T06:59:43.982 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.70848.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43279.log 2024-09-12T06:59:43.983 INFO:teuthology.orchestra.run.smithi152.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.70848.log.gz 2024-09-12T06:59:43.983 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.86390.log: gzip 86.9% -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67992.log 2024-09-12T06:59:43.983 INFO:teuthology.orchestra.run.smithi152.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.86390.log.gz 2024-09-12T06:59:43.983 INFO:teuthology.orchestra.run.smithi152.stderr: 91.7% -- replaced with /var/log/ceph/cluster1-client.mirror.73161.log.gz 2024-09-12T06:59:43.983 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83976.log 2024-09-12T06:59:43.983 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.43279.log: /var/log/ceph/cluster2-client.mirror.67992.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40262.log 2024-09-12T06:59:43.983 INFO:teuthology.orchestra.run.smithi152.stderr: 87.8%/var/log/ceph/cluster2-client.mirror.83976.log: -- replaced with /var/log/ceph/cluster2-client.mirror.67992.log.gz 2024-09-12T06:59:43.983 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45801.log 2024-09-12T06:59:43.983 INFO:teuthology.orchestra.run.smithi152.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.83976.log.gz 2024-09-12T06:59:43.983 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.40262.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41616.log 2024-09-12T06:59:43.983 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.45801.log: 93.1% -- replaced with /var/log/ceph/cluster2-client.mirror.43464.log.gz 2024-09-12T06:59:43.983 INFO:teuthology.orchestra.run.smithi152.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.43279.log.gz 2024-09-12T06:59:43.983 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47110.log 2024-09-12T06:59:43.983 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.41616.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43948.log 2024-09-12T06:59:43.984 INFO:teuthology.orchestra.run.smithi152.stderr:gzip/var/log/ceph/cluster2-client.mirror.47110.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81090.log 2024-09-12T06:59:43.984 INFO:teuthology.orchestra.run.smithi152.stderr: 84.3%/var/log/ceph/cluster2-client.mirror.43948.log: -- replaced with /var/log/ceph/cluster2-client.mirror.47110.log.gz 2024-09-12T06:59:43.984 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88098.log 2024-09-12T06:59:43.984 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.81090.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81119.log 2024-09-12T06:59:43.984 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.88098.log: 83.6% -- replaced with /var/log/ceph/cluster2-client.mirror.88098.log.gz 2024-09-12T06:59:43.984 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86322.log 2024-09-12T06:59:43.984 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.81119.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39605.log 2024-09-12T06:59:43.984 INFO:teuthology.orchestra.run.smithi152.stderr: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.81090.log.gz 2024-09-12T06:59:43.984 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.86322.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.86322.log.gz 2024-09-12T06:59:43.984 INFO:teuthology.orchestra.run.smithi152.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.81119.log.gz 2024-09-12T06:59:43.984 INFO:teuthology.orchestra.run.smithi152.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.43948.log.gz 2024-09-12T06:59:43.984 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37498.log 2024-09-12T06:59:43.984 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.39605.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.86592.log 2024-09-12T06:59:43.985 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.37498.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87704.log 2024-09-12T06:59:43.985 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.0.86592.log: gzip -5 20.8% -- replaced with /var/log/ceph/cluster1-client.mirror.0.86592.log.gz 2024-09-12T06:59:43.985 INFO:teuthology.orchestra.run.smithi152.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.83567.log 2024-09-12T06:59:43.985 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.87704.log: gzip -5 --verbose -- 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.87704.log.gz 2024-09-12T06:59:43.985 INFO:teuthology.orchestra.run.smithi152.stderr: /var/log/ceph/cluster1-client.mirror.60886.log 2024-09-12T06:59:43.985 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.83567.log: 68.1%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69226.log 2024-09-12T06:59:43.985 INFO:teuthology.orchestra.run.smithi152.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.37498.log.gz 2024-09-12T06:59:43.985 INFO:teuthology.orchestra.run.smithi152.stderr:gzip/var/log/ceph/cluster1-client.mirror.60886.log: -5 --verbose -- 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.83567.log.gz 2024-09-12T06:59:43.985 INFO:teuthology.orchestra.run.smithi152.stderr: /var/log/ceph/cluster2-client.mirror.37805.log 2024-09-12T06:59:43.985 INFO:teuthology.orchestra.run.smithi152.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.60886.log.gz 2024-09-12T06:59:43.985 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76962.log 2024-09-12T06:59:43.985 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.37805.log: /var/log/ceph/cluster2-client.mirror.69226.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85220.log 2024-09-12T06:59:43.985 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.76962.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76240.log 2024-09-12T06:59:43.985 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.85220.log: 86.9% 54.2% -- replaced with /var/log/ceph/cluster2-client.mirror.37805.log.gz 2024-09-12T06:59:43.986 INFO:teuthology.orchestra.run.smithi152.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.85220.log.gz 2024-09-12T06:59:43.986 INFO:teuthology.orchestra.run.smithi152.stderr: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.69226.log.gz 2024-09-12T06:59:43.986 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.76962.log.gz 2024-09-12T06:59:43.986 INFO:teuthology.orchestra.run.smithi152.stderr: --verbose -- /var/log/ceph/cluster2-client.admin.90539.log 2024-09-12T06:59:43.986 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85940.log/var/log/ceph/cluster2-client.mirror.76240.log: 2024-09-12T06:59:43.986 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.admin.90539.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.90539.log.gz 2024-09-12T06:59:43.993 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.85940.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.85940.log.gz 2024-09-12T06:59:43.993 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83018.log 2024-09-12T06:59:43.993 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89136.log 2024-09-12T06:59:43.994 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.83018.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79248.log 2024-09-12T06:59:43.995 INFO:teuthology.orchestra.run.smithi152.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.83018.log.gz 2024-09-12T06:59:43.995 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.89136.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.89136.log.gz 2024-09-12T06:59:43.998 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56672.log 2024-09-12T06:59:43.998 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.79248.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.89815.log 2024-09-12T06:59:44.001 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.56672.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.79248.log.gz 2024-09-12T06:59:44.001 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36082.log 2024-09-12T06:59:44.002 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.admin.89815.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.89815.log.gz 2024-09-12T06:59:44.002 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46712.log 2024-09-12T06:59:44.002 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.admin.36082.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36082.log.gz 2024-09-12T06:59:44.002 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72162.log 2024-09-12T06:59:44.002 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.46712.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78845.log 2024-09-12T06:59:44.006 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.72162.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.72162.log.gz 2024-09-12T06:59:44.006 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82684.log 2024-09-12T06:59:44.006 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.78845.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.78845.log.gz 2024-09-12T06:59:44.006 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41080.log 2024-09-12T06:59:44.006 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.82684.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.82684.log.gz 2024-09-12T06:59:44.006 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54770.log 2024-09-12T06:59:44.006 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.41080.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83812.log 2024-09-12T06:59:44.013 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.54770.log: 84.3% -- replaced with /var/log/ceph/cluster2-client.mirror.46712.log.gz 2024-09-12T06:59:44.014 INFO:teuthology.orchestra.run.smithi096.stderr: 25.3% -- replaced with /var/log/ceph/cluster1-client.admin.32374.log.gz 2024-09-12T06:59:44.015 INFO:teuthology.orchestra.run.smithi096.stderr: 94.1% -- replaced with /var/log/ceph/cluster1.audit.log.gz 2024-09-12T06:59:44.018 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85535.log 2024-09-12T06:59:44.019 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.83812.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.83812.log.gz 2024-09-12T06:59:44.031 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44946.log 2024-09-12T06:59:44.031 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.85535.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.85535.log.gz 2024-09-12T06:59:44.043 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64453.log 2024-09-12T06:59:44.057 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.44946.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75831.log 2024-09-12T06:59:44.058 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.64453.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.64453.log.gz 2024-09-12T06:59:44.065 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86210.log 2024-09-12T06:59:44.070 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.75831.log: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.40262.log.gz 2024-09-12T06:59:44.078 INFO:teuthology.orchestra.run.smithi152.stderr: 91.8% -- replaced with /var/log/ceph/cluster1-client.mirror.75831.log.gz 2024-09-12T06:59:44.079 INFO:teuthology.orchestra.run.smithi152.stderr: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.69654.log.gz 2024-09-12T06:59:44.080 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87910.log 2024-09-12T06:59:44.081 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.86210.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.86210.log.gz 2024-09-12T06:59:44.081 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63017.log 2024-09-12T06:59:44.082 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.87910.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.87910.log.gz 2024-09-12T06:59:44.082 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71234.log 2024-09-12T06:59:44.084 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.63017.log: 91.9% -- replaced with /var/log/ceph/cluster2-client.mirror.63017.log.gz 2024-09-12T06:59:44.086 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81303.log 2024-09-12T06:59:44.087 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.71234.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.71234.log.gz 2024-09-12T06:59:44.087 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44266.log 2024-09-12T06:59:44.094 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.81303.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.81303.log.gz 2024-09-12T06:59:44.094 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52042.log 2024-09-12T06:59:44.096 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.44266.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81374.log 2024-09-12T06:59:44.096 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.52042.log: 92.9% -- replaced with /var/log/ceph/cluster2-client.mirror.40968.log.gz 2024-09-12T06:59:44.097 INFO:teuthology.orchestra.run.smithi152.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.44266.log.gz 2024-09-12T06:59:44.108 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88842.log 2024-09-12T06:59:44.110 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.81374.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.81374.log.gz 2024-09-12T06:59:44.139 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84079.log 2024-09-12T06:59:44.140 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.88842.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.44371.log.gz 2024-09-12T06:59:44.141 INFO:teuthology.orchestra.run.smithi152.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.88842.log.gz 2024-09-12T06:59:44.148 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81261.log 2024-09-12T06:59:44.163 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.84079.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.84079.log.gz 2024-09-12T06:59:44.167 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84297.log 2024-09-12T06:59:44.172 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.81261.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.81261.log.gz 2024-09-12T06:59:44.183 INFO:teuthology.orchestra.run.smithi152.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.56672.log.gz 2024-09-12T06:59:44.184 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49848.log 2024-09-12T06:59:44.185 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.84297.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.84297.log.gz 2024-09-12T06:59:44.185 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60433.log 2024-09-12T06:59:44.197 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.49848.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39493.log 2024-09-12T06:59:44.197 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.60433.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.60433.log.gz 2024-09-12T06:59:44.202 INFO:teuthology.orchestra.run.smithi152.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.40331.log.gz 2024-09-12T06:59:44.204 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69134.log 2024-09-12T06:59:44.213 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.39493.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89216.log 2024-09-12T06:59:44.213 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.69134.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.69134.log.gz 2024-09-12T06:59:44.220 INFO:teuthology.orchestra.run.smithi152.stderr: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.76240.log.gz 2024-09-12T06:59:44.223 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43871.log 2024-09-12T06:59:44.223 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.89216.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.89216.log.gz 2024-09-12T06:59:44.223 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.89743.log 2024-09-12T06:59:44.224 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.43871.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81283.log 2024-09-12T06:59:44.224 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.admin.89743.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.89743.log.gz 2024-09-12T06:59:44.224 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47420.log 2024-09-12T06:59:44.225 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.81283.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.81283.log.gz 2024-09-12T06:59:44.225 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87647.log 2024-09-12T06:59:44.226 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.47420.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.43871.log.gz 2024-09-12T06:59:44.250 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89597.log 2024-09-12T06:59:44.251 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.87647.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.87647.log.gz 2024-09-12T06:59:44.251 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45627.log 2024-09-12T06:59:44.251 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.89597.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.89597.log.gz 2024-09-12T06:59:44.258 INFO:teuthology.orchestra.run.smithi152.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48735.log.gz 2024-09-12T06:59:44.258 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72095.log 2024-09-12T06:59:44.259 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.45627.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76198.log 2024-09-12T06:59:44.260 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.72095.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.72095.log.gz 2024-09-12T06:59:44.260 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.27926.log 2024-09-12T06:59:44.263 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.76198.log: 91.9% -- replaced with /var/log/ceph/cluster2-client.mirror.76198.log.gz 2024-09-12T06:59:44.263 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78912.log 2024-09-12T06:59:44.264 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph.tmp-client.admin.27926.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.27926.log.gz 2024-09-12T06:59:44.264 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75067.log 2024-09-12T06:59:44.278 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.78912.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85310.log 2024-09-12T06:59:44.279 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.75067.log: 91.8% -- replaced with /var/log/ceph/cluster1-client.mirror.75067.log.gz 2024-09-12T06:59:44.293 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60055.log 2024-09-12T06:59:44.293 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.85310.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.85310.log.gz 2024-09-12T06:59:44.299 INFO:teuthology.orchestra.run.smithi152.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.67362.log.gz 2024-09-12T06:59:44.300 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47875.log 2024-09-12T06:59:44.301 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.60055.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.60055.log.gz 2024-09-12T06:59:44.301 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46076.log 2024-09-12T06:59:44.302 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.47875.log: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47875.log.gz 2024-09-12T06:59:44.303 INFO:teuthology.orchestra.run.smithi152.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48976.log.gz 2024-09-12T06:59:44.303 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40876.log 2024-09-12T06:59:44.305 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.46076.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84179.log 2024-09-12T06:59:44.305 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.40876.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63724.log 2024-09-12T06:59:44.305 INFO:teuthology.orchestra.run.smithi152.stderr:gzip/var/log/ceph/cluster2-client.mirror.84179.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37941.log 2024-09-12T06:59:44.305 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.63724.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.84179.log.gz 2024-09-12T06:59:44.306 INFO:teuthology.orchestra.run.smithi152.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.40876.log.gz 2024-09-12T06:59:44.306 INFO:teuthology.orchestra.run.smithi152.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.63724.log.gz 2024-09-12T06:59:44.307 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87357.log 2024-09-12T06:59:44.308 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.37941.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79156.log 2024-09-12T06:59:44.310 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.87357.log: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.88776.log.gz 2024-09-12T06:59:44.311 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37399.log 2024-09-12T06:59:44.311 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.79156.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87071.log 2024-09-12T06:59:44.313 INFO:teuthology.orchestra.run.smithi152.stderr: 88.0%/var/log/ceph/cluster1-client.admin.37399.log: -- replaced with /var/log/ceph/cluster2-client.mirror.79156.log.gz 2024-09-12T06:59:44.314 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37399.log.gz 2024-09-12T06:59:44.314 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79773.log 2024-09-12T06:59:44.314 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.87071.log: -- /var/log/ceph/cluster1-client.mirror.63801.log 2024-09-12T06:59:44.314 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.79773.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.79773.log.gz 2024-09-12T06:59:44.330 INFO:teuthology.orchestra.run.smithi152.stderr:gzip 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.87071.log.gz 2024-09-12T06:59:44.330 INFO:teuthology.orchestra.run.smithi152.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56119.log 2024-09-12T06:59:44.330 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84567.log 2024-09-12T06:59:44.331 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.56119.log: /var/log/ceph/cluster1-client.mirror.63801.log: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56119.log.gz 2024-09-12T06:59:44.331 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36084.log 2024-09-12T06:59:44.331 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.84567.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.84567.log.gz 2024-09-12T06:59:44.331 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81861.log 2024-09-12T06:59:44.331 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.admin.36084.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36084.log.gz 2024-09-12T06:59:44.331 INFO:teuthology.orchestra.run.smithi152.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-client.mirror.63801.log.gz 2024-09-12T06:59:44.331 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70377.log 2024-09-12T06:59:44.331 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.81861.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40289.log 2024-09-12T06:59:44.331 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.70377.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.70377.log.gz 2024-09-12T06:59:44.331 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67641.log 2024-09-12T06:59:44.331 INFO:teuthology.orchestra.run.smithi152.stderr: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.81861.log.gz 2024-09-12T06:59:44.331 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.40289.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59677.log 2024-09-12T06:59:44.331 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.67641.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38659.log 2024-09-12T06:59:44.332 INFO:teuthology.orchestra.run.smithi152.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.67641.log.gz 2024-09-12T06:59:44.332 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.59677.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62992.log 2024-09-12T06:59:44.332 INFO:teuthology.orchestra.run.smithi152.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.59677.log.gz 2024-09-12T06:59:44.332 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.38659.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88897.log 2024-09-12T06:59:44.332 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.62992.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73561.log 2024-09-12T06:59:44.332 INFO:teuthology.orchestra.run.smithi152.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.62992.log.gz 2024-09-12T06:59:44.332 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.73561.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73494.log 2024-09-12T06:59:44.332 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.88897.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.88897.log.gz 2024-09-12T06:59:44.332 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82286.log 2024-09-12T06:59:44.332 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.73494.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.73494.log.gz 2024-09-12T06:59:44.332 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89237.log 2024-09-12T06:59:44.332 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.82286.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.82286.log.gz 2024-09-12T06:59:44.332 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68104.log 2024-09-12T06:59:44.333 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.89237.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.89237.log.gz 2024-09-12T06:59:44.333 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64587.log 2024-09-12T06:59:44.334 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.68104.log: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68104.log.gz 2024-09-12T06:59:44.334 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72130.log 2024-09-12T06:59:44.343 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.64587.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70469.log 2024-09-12T06:59:44.344 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.72130.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.72130.log.gz 2024-09-12T06:59:44.349 INFO:teuthology.orchestra.run.smithi152.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49848.log.gz 2024-09-12T06:59:44.352 INFO:teuthology.orchestra.run.smithi152.stderr: 95.2% -- replaced with /var/log/ceph/cluster2-client.mirror.87357.log.gz 2024-09-12T06:59:44.352 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38617.log 2024-09-12T06:59:44.355 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.70469.log: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.70469.log.gz 2024-09-12T06:59:44.355 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53693.log 2024-09-12T06:59:44.356 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.38617.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39560.log 2024-09-12T06:59:44.358 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.53693.log: 84.4% -- replaced with /var/log/ceph/cluster2-client.mirror.53693.log.gz 2024-09-12T06:59:44.358 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89003.log 2024-09-12T06:59:44.358 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.39560.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55606.log 2024-09-12T06:59:44.359 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.89003.log: 83.4% -- replaced with /var/log/ceph/cluster2-client.mirror.89003.log.gz 2024-09-12T06:59:44.359 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39137.log 2024-09-12T06:59:44.377 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.55606.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70026.log 2024-09-12T06:59:44.378 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.39137.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.39493.log.gz 2024-09-12T06:59:44.379 INFO:teuthology.orchestra.run.smithi152.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52042.log.gz 2024-09-12T06:59:44.381 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42286.log 2024-09-12T06:59:44.381 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.70026.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.70026.log.gz 2024-09-12T06:59:44.382 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88077.log 2024-09-12T06:59:44.385 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.42286.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87572.log 2024-09-12T06:59:44.388 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.88077.log: 91.7% -- replaced with /var/log/ceph/cluster2-client.mirror.88077.log.gz 2024-09-12T06:59:44.388 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83925.log 2024-09-12T06:59:44.389 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.87572.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.42286.log.gz 2024-09-12T06:59:44.401 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72566.log 2024-09-12T06:59:44.403 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.83925.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.83925.log.gz 2024-09-12T06:59:44.416 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37447.log 2024-09-12T06:59:44.417 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.72566.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.72566.log.gz 2024-09-12T06:59:44.428 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70960.log 2024-09-12T06:59:44.442 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.admin.37447.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.40289.log.gz 2024-09-12T06:59:44.442 INFO:teuthology.orchestra.run.smithi152.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.45627.log.gz 2024-09-12T06:59:44.442 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48595.log 2024-09-12T06:59:44.443 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.70960.log: 80.1% -- replaced with /var/log/ceph/cluster1-client.admin.37447.log.gz 2024-09-12T06:59:44.445 INFO:teuthology.orchestra.run.smithi152.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-client.mirror.70960.log.gz 2024-09-12T06:59:44.456 INFO:teuthology.orchestra.run.smithi152.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.38617.log.gz 2024-09-12T06:59:44.459 INFO:teuthology.orchestra.run.smithi152.stderr: 93.1% -- replaced with /var/log/ceph/cluster2-client.mirror.44946.log.gz 2024-09-12T06:59:44.461 INFO:teuthology.orchestra.run.smithi152.stderr: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.73561.log.gz 2024-09-12T06:59:44.462 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35744.log 2024-09-12T06:59:44.475 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.48595.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86874.log 2024-09-12T06:59:44.476 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.admin.35744.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35744.log.gz 2024-09-12T06:59:44.479 INFO:teuthology.orchestra.run.smithi152.stderr: 95.9% -- replaced with /var/log/ceph/cluster2-client.mirror.54770.log.gz 2024-09-12T06:59:44.488 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76576.log 2024-09-12T06:59:44.490 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.86874.log: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.86874.log.gz 2024-09-12T06:59:44.491 INFO:teuthology.orchestra.run.smithi152.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.39560.log.gz 2024-09-12T06:59:44.503 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66545.log 2024-09-12T06:59:44.508 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.76576.log: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.76576.log.gz 2024-09-12T06:59:44.509 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86789.log 2024-09-12T06:59:44.510 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.66545.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.66545.log.gz 2024-09-12T06:59:44.510 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59712.log 2024-09-12T06:59:44.512 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.86789.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.86789.log.gz 2024-09-12T06:59:44.513 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42193.log 2024-09-12T06:59:44.514 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.59712.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.59712.log.gz 2024-09-12T06:59:44.514 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88647.log 2024-09-12T06:59:44.515 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.42193.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44524.log 2024-09-12T06:59:44.518 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.88647.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.88647.log.gz 2024-09-12T06:59:44.519 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87484.log 2024-09-12T06:59:44.519 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.44524.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79290.log 2024-09-12T06:59:44.532 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.87484.log: 91.7% -- replaced with /var/log/ceph/cluster2-client.mirror.87484.log.gz 2024-09-12T06:59:44.533 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45342.log 2024-09-12T06:59:44.542 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.79290.log: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.79290.log.gz 2024-09-12T06:59:44.542 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81606.log 2024-09-12T06:59:44.543 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.45342.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83518.log 2024-09-12T06:59:44.546 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.81606.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.81606.log.gz 2024-09-12T06:59:44.546 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89278.log 2024-09-12T06:59:44.547 INFO:teuthology.orchestra.run.smithi152.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45342.log.gz 2024-09-12T06:59:44.548 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.83518.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.83518.log.gz 2024-09-12T06:59:44.548 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88534.log 2024-09-12T06:59:44.549 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.89278.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.42193.log.gz 2024-09-12T06:59:44.550 INFO:teuthology.orchestra.run.smithi152.stderr: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.89278.log.gz 2024-09-12T06:59:44.551 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81790.log 2024-09-12T06:59:44.551 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.88534.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.88534.log.gz 2024-09-12T06:59:44.555 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89318.log 2024-09-12T06:59:44.558 INFO:teuthology.orchestra.run.smithi096.stderr: 91.2% -- replaced with /var/log/ceph/cluster2-mgr.x.log.gz 2024-09-12T06:59:44.560 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.81790.log: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.78912.log.gz 2024-09-12T06:59:44.561 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36632.log 2024-09-12T06:59:44.561 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.89318.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.89318.log.gz 2024-09-12T06:59:44.566 INFO:teuthology.orchestra.run.smithi152.stderr: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.81790.log.gz 2024-09-12T06:59:44.566 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78080.log 2024-09-12T06:59:44.566 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.admin.36632.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75453.log 2024-09-12T06:59:44.566 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.78080.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36632.log.gz 2024-09-12T06:59:44.566 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88815.log 2024-09-12T06:59:44.566 INFO:teuthology.orchestra.run.smithi152.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78080.log.gz 2024-09-12T06:59:44.566 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.75453.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45009.log 2024-09-12T06:59:44.566 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.88815.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.88815.log.gz 2024-09-12T06:59:44.566 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88688.log 2024-09-12T06:59:44.566 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.45009.log: 91.7% -- replaced with /var/log/ceph/cluster1-client.mirror.75453.log.gz 2024-09-12T06:59:44.567 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73462.log 2024-09-12T06:59:44.570 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.88688.log: 91.7% -- replaced with /var/log/ceph/cluster2-client.mirror.88688.log.gz 2024-09-12T06:59:44.571 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76551.log 2024-09-12T06:59:44.571 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72987.log 2024-09-12T06:59:44.571 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.76551.log: /var/log/ceph/cluster2-client.mirror.73462.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.73462.log.gz 2024-09-12T06:59:44.571 INFO:teuthology.orchestra.run.smithi152.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.76551.log.gz 2024-09-12T06:59:44.571 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87672.log 2024-09-12T06:59:44.573 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.72987.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.72987.log.gz 2024-09-12T06:59:44.573 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88668.log 2024-09-12T06:59:44.573 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.87672.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74283.log 2024-09-12T06:59:44.574 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.88668.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.88668.log.gz 2024-09-12T06:59:44.574 INFO:teuthology.orchestra.run.smithi152.stderr: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.87672.log.gz 2024-09-12T06:59:44.574 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.86583.log 2024-09-12T06:59:44.576 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.74283.log: 92.1% -- replaced with /var/log/ceph/cluster2-client.mirror.74283.log.gz 2024-09-12T06:59:44.580 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86030.log 2024-09-12T06:59:44.581 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.0.86583.log: 19.3% -- replaced with /var/log/ceph/cluster1-client.mirror.0.86583.log.gz 2024-09-12T06:59:44.581 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68437.log 2024-09-12T06:59:44.581 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.86030.log: gzip -5 --verbose -- 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.86030.log.gz 2024-09-12T06:59:44.581 INFO:teuthology.orchestra.run.smithi152.stderr: /var/log/ceph/cluster2-client.mirror.81212.log 2024-09-12T06:59:44.581 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44173.log 2024-09-12T06:59:44.581 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.81212.log: /var/log/ceph/cluster1-client.mirror.68437.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.81212.log.gz 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68437.log.gz 2024-09-12T06:59:44.581 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79680.log 2024-09-12T06:59:44.581 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.44173.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82520.log 2024-09-12T06:59:44.589 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.79680.log: 2024-09-12T06:59:44.590 INFO:teuthology.orchestra.run.smithi152.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.44173.log.gz 2024-09-12T06:59:44.590 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80776.log 2024-09-12T06:59:44.590 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.82520.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.79680.log.gz 2024-09-12T06:59:44.590 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42437.log 2024-09-12T06:59:44.590 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.80776.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.80776.log.gz 2024-09-12T06:59:44.590 INFO:teuthology.orchestra.run.smithi152.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44524.log.gz 2024-09-12T06:59:44.590 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73116.log 2024-09-12T06:59:44.590 INFO:teuthology.orchestra.run.smithi152.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.82520.log.gz 2024-09-12T06:59:44.590 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.42437.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44567.log 2024-09-12T06:59:44.590 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.73116.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74226.log 2024-09-12T06:59:44.590 INFO:teuthology.orchestra.run.smithi152.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.73116.log.gz 2024-09-12T06:59:44.590 INFO:teuthology.orchestra.run.smithi152.stderr: 88.7%gzip/var/log/ceph/cluster2-client.mirror.74226.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39114.log 2024-09-12T06:59:44.590 INFO:teuthology.orchestra.run.smithi152.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.42437.log.gz 2024-09-12T06:59:44.591 INFO:teuthology.orchestra.run.smithi152.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.74226.log.gz 2024-09-12T06:59:44.591 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.44567.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65217.log 2024-09-12T06:59:44.591 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.39114.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59744.log 2024-09-12T06:59:44.591 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70412.log 2024-09-12T06:59:44.591 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.65217.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.65217.log.gz 2024-09-12T06:59:44.591 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.59744.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.59744.log.gz 2024-09-12T06:59:44.594 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.70412.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38419.log 2024-09-12T06:59:44.594 INFO:teuthology.orchestra.run.smithi152.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.70412.log.gz 2024-09-12T06:59:44.594 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.90457.log 2024-09-12T06:59:44.594 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81232.log 2024-09-12T06:59:44.594 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.admin.90457.log: /var/log/ceph/cluster1-client.admin.38419.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.90457.log.gz 2024-09-12T06:59:44.594 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77969.log 2024-09-12T06:59:44.595 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.81232.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.81232.log.gz 2024-09-12T06:59:44.597 INFO:teuthology.orchestra.run.smithi152.stderr:gzip/var/log/ceph/cluster2-client.mirror.77969.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49876.log 2024-09-12T06:59:44.598 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44813.log 2024-09-12T06:59:44.598 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.49876.log: 84.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49876.log.gz 2024-09-12T06:59:44.600 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85805.log 2024-09-12T06:59:44.600 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.44813.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68462.log 2024-09-12T06:59:44.600 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.85805.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.85805.log.gz 2024-09-12T06:59:44.601 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46169.log 2024-09-12T06:59:44.603 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.68462.log: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.68462.log.gz 2024-09-12T06:59:44.603 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80056.log 2024-09-12T06:59:44.613 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.46169.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.37897.log 2024-09-12T06:59:44.613 INFO:teuthology.orchestra.run.smithi152.stderr: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.64587.log.gz 2024-09-12T06:59:44.614 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.80056.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46146.log 2024-09-12T06:59:44.614 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.0.37897.log: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48595.log.gz 2024-09-12T06:59:44.614 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85557.log 2024-09-12T06:59:44.614 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.46146.log: -- /var/log/ceph/cluster1-client.mirror.84905.log 2024-09-12T06:59:44.614 INFO:teuthology.orchestra.run.smithi152.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.38419.log.gz 2024-09-12T06:59:44.614 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64951.log 2024-09-12T06:59:44.614 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.84905.log: 96.9% -- replaced with /var/log/ceph/cluster2-client.mirror.77969.log.gz 2024-09-12T06:59:44.614 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.85557.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.80056.log.gz 2024-09-12T06:59:44.614 INFO:teuthology.orchestra.run.smithi152.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.85557.log.gz 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.84905.log.gz 2024-09-12T06:59:44.614 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-09-12T06:59:44.614 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66479.log 2024-09-12T06:59:44.614 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.64951.log: 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46146.log.gz 2024-09-12T06:59:44.614 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86300.log 2024-09-12T06:59:44.615 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.66479.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39720.log 2024-09-12T06:59:44.615 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.86300.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60525.log 2024-09-12T06:59:44.615 INFO:teuthology.orchestra.run.smithi152.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.86300.log.gz 2024-09-12T06:59:44.615 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.39720.log: 91.4% -- replaced with /var/log/ceph/cluster1-client.mirror.64951.log.gz 2024-09-12T06:59:44.615 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79722.log 2024-09-12T06:59:44.615 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.60525.log: 91.6%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44231.log 2024-09-12T06:59:44.615 INFO:teuthology.orchestra.run.smithi152.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.66479.log.gz 2024-09-12T06:59:44.615 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.79722.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70940.log 2024-09-12T06:59:44.615 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68059.log 2024-09-12T06:59:44.615 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.70940.log: /var/log/ceph/cluster2-client.mirror.44231.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40548.log 2024-09-12T06:59:44.615 INFO:teuthology.orchestra.run.smithi152.stderr: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.60525.log.gz 2024-09-12T06:59:44.615 INFO:teuthology.orchestra.run.smithi152.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.44231.log.gz/var/log/ceph/cluster1-client.mirror.68059.log: 2024-09-12T06:59:44.615 INFO:teuthology.orchestra.run.smithi152.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68059.log.gz 2024-09-12T06:59:44.616 INFO:teuthology.orchestra.run.smithi152.stderr: 91.8% -- replaced with /var/log/ceph/cluster2-client.mirror.70940.log.gz 2024-09-12T06:59:44.616 INFO:teuthology.orchestra.run.smithi152.stderr: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.79722.log.gz 2024-09-12T06:59:44.616 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68027.log 2024-09-12T06:59:44.623 INFO:teuthology.orchestra.run.smithi096.stderr: 91.3% -- replaced with /var/log/ceph/cluster1-mgr.x.log.gz 2024-09-12T06:59:44.656 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.40548.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73427.log 2024-09-12T06:59:44.657 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.68027.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45300.log 2024-09-12T06:59:44.657 INFO:teuthology.orchestra.run.smithi152.stderr: 87.9%/var/log/ceph/cluster2-client.mirror.73427.log: gzip -- replaced with /var/log/ceph/cluster2-client.mirror.68027.log.gz 2024-09-12T06:59:44.657 INFO:teuthology.orchestra.run.smithi152.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88406.log 2024-09-12T06:59:44.657 INFO:teuthology.orchestra.run.smithi152.stderr: 87.8%/var/log/ceph/cluster2-client.mirror.45300.log: -- replaced with /var/log/ceph/cluster2-client.mirror.73427.log.gz 2024-09-12T06:59:44.657 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41914.log 2024-09-12T06:59:44.657 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.88406.log: 83.3% -- replaced with /var/log/ceph/cluster2-client.mirror.88406.log.gz 2024-09-12T06:59:44.657 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.40548.log.gz 2024-09-12T06:59:44.657 INFO:teuthology.orchestra.run.smithi152.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.39969.log 2024-09-12T06:59:44.657 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.41914.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45169.log 2024-09-12T06:59:44.657 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.39969.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78813.log 2024-09-12T06:59:44.657 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.45169.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.41914.log.gz 2024-09-12T06:59:44.657 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66048.log 2024-09-12T06:59:44.658 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78013.log 2024-09-12T06:59:44.658 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.78813.log: /var/log/ceph/cluster1-client.mirror.66048.log: 88.1%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85332.log 2024-09-12T06:59:44.658 INFO:teuthology.orchestra.run.smithi152.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.78813.log.gz 2024-09-12T06:59:44.658 INFO:teuthology.orchestra.run.smithi152.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.66048.log.gz 2024-09-12T06:59:44.658 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.78013.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62253.log 2024-09-12T06:59:44.658 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.85332.log: 87.9%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82491.log 2024-09-12T06:59:44.658 INFO:teuthology.orchestra.run.smithi152.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.78013.log.gz 2024-09-12T06:59:44.658 INFO:teuthology.orchestra.run.smithi152.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.85332.log.gz 2024-09-12T06:59:44.658 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.62253.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81819.log 2024-09-12T06:59:44.658 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.82491.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44637.log 2024-09-12T06:59:44.658 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.81819.log: 92.2% 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.62253.log.gz 2024-09-12T06:59:44.658 INFO:teuthology.orchestra.run.smithi152.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.81819.log.gz 2024-09-12T06:59:44.658 INFO:teuthology.orchestra.run.smithi152.stderr: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.82491.log.gz 2024-09-12T06:59:44.659 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.90163.log 2024-09-12T06:59:44.659 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.44637.log: -- /var/log/ceph/cluster1-client.mirror.84972.log 2024-09-12T06:59:44.659 INFO:teuthology.orchestra.run.smithi152.stderr:gzip/var/log/ceph/cluster2-client.admin.90163.log: -5 --verbose -- 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.90163.log.gz 2024-09-12T06:59:44.659 INFO:teuthology.orchestra.run.smithi152.stderr: /var/log/ceph/cluster1-client.mirror.86052.log 2024-09-12T06:59:44.659 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.84972.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70058.log 2024-09-12T06:59:44.659 INFO:teuthology.orchestra.run.smithi152.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.84972.log.gz 2024-09-12T06:59:44.659 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.86052.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.86052.log.gz 2024-09-12T06:59:44.659 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52754.log 2024-09-12T06:59:44.659 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.70058.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43675.log 2024-09-12T06:59:44.659 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.52754.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70058.log.gz 2024-09-12T06:59:44.659 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43407.log 2024-09-12T06:59:44.659 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87962.log 2024-09-12T06:59:44.660 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.43675.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37052.log 2024-09-12T06:59:44.660 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.43407.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82775.log/var/log/ceph/cluster2-client.mirror.87962.log: 2024-09-12T06:59:44.660 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.admin.37052.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37052.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65603.log 2024-09-12T06:59:44.660 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-09-12T06:59:44.660 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81628.log 2024-09-12T06:59:44.660 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.65603.log: /var/log/ceph/cluster2-client.mirror.82775.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.65603.log.gz 2024-09-12T06:59:44.660 INFO:teuthology.orchestra.run.smithi152.stderr: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.87962.log.gz 2024-09-12T06:59:44.660 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39343.log 2024-09-12T06:59:44.660 INFO:teuthology.orchestra.run.smithi152.stderr: 90.7%/var/log/ceph/cluster2-client.mirror.81628.log: -- replaced with /var/log/ceph/cluster2-client.mirror.82775.log.gz 2024-09-12T06:59:44.660 INFO:teuthology.orchestra.run.smithi152.stderr: 88.1%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44723.log 2024-09-12T06:59:44.660 INFO:teuthology.orchestra.run.smithi152.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.81628.log.gz 2024-09-12T06:59:44.660 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37464.log 2024-09-12T06:59:44.661 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.39343.log: /var/log/ceph/cluster2-client.mirror.44723.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.43407.log.gz 2024-09-12T06:59:44.661 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71368.log 2024-09-12T06:59:44.661 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77561.log 2024-09-12T06:59:44.661 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.admin.37464.log: gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.71368.log: -- /var/log/ceph/cluster1-client.mirror.79268.log 2024-09-12T06:59:44.661 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.77561.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79542.log 2024-09-12T06:59:44.661 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.79268.log: 91.5% -- replaced with /var/log/ceph/cluster1-client.mirror.77561.log.gz 2024-09-12T06:59:44.661 INFO:teuthology.orchestra.run.smithi152.stderr: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.79268.log.gz 2024-09-12T06:59:44.661 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42902.log 2024-09-12T06:59:44.661 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.79542.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40580.log 2024-09-12T06:59:44.661 INFO:teuthology.orchestra.run.smithi152.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.79542.log.gz 2024-09-12T06:59:44.661 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.42902.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80867.log 2024-09-12T06:59:44.661 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39401.log 2024-09-12T06:59:44.661 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.40580.log: gzip/var/log/ceph/cluster2-client.mirror.80867.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74955.log 2024-09-12T06:59:44.662 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.39401.log: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.80867.log.gz 2024-09-12T06:59:44.662 INFO:teuthology.orchestra.run.smithi152.stderr: 80.1% -- replaced with /var/log/ceph/cluster1-client.admin.37464.log.gz 2024-09-12T06:59:44.662 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52019.log 2024-09-12T06:59:44.662 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.74955.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.74955.log.gz 2024-09-12T06:59:44.662 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53716.log 2024-09-12T06:59:44.662 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.52019.log: 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52019.log.gz 2024-09-12T06:59:44.662 INFO:teuthology.orchestra.run.smithi152.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47420.log.gz 2024-09-12T06:59:44.662 INFO:teuthology.orchestra.run.smithi152.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.39343.log.gz 2024-09-12T06:59:44.662 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68890.log 2024-09-12T06:59:44.662 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.53716.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76937.log 2024-09-12T06:59:44.662 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.68890.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44052.log 2024-09-12T06:59:44.662 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.76937.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.76937.log.gz 2024-09-12T06:59:44.662 INFO:teuthology.orchestra.run.smithi152.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.45300.log.gz 2024-09-12T06:59:44.662 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39224.log 2024-09-12T06:59:44.663 INFO:teuthology.orchestra.run.smithi152.stderr: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.40580.log.gz 2024-09-12T06:59:44.663 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.44052.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43484.log 2024-09-12T06:59:44.663 INFO:teuthology.orchestra.run.smithi152.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.42902.log.gz 2024-09-12T06:59:44.663 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.39224.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83854.log 2024-09-12T06:59:44.663 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.43484.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88245.log 2024-09-12T06:59:44.663 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.83854.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.39401.log.gz 2024-09-12T06:59:44.663 INFO:teuthology.orchestra.run.smithi152.stderr: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.83854.log.gz 2024-09-12T06:59:44.663 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45778.log 2024-09-12T06:59:44.663 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.88245.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.88245.log.gz 2024-09-12T06:59:44.663 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.90205.log 2024-09-12T06:59:44.663 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.45778.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73813.log 2024-09-12T06:59:44.663 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.admin.90205.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.90205.log.gz 2024-09-12T06:59:44.663 INFO:teuthology.orchestra.run.smithi152.stderr: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45778.log.gz 2024-09-12T06:59:44.664 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87277.log 2024-09-12T06:59:44.664 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.73813.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.73813.log.gz 2024-09-12T06:59:44.664 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88941.log 2024-09-12T06:59:44.665 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.87277.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85602.log 2024-09-12T06:59:44.672 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.88941.log: 88.3%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83883.log 2024-09-12T06:59:44.673 INFO:teuthology.orchestra.run.smithi152.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.88941.log.gz 2024-09-12T06:59:44.684 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.85602.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.85602.log.gz 2024-09-12T06:59:44.685 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.90079.log 2024-09-12T06:59:44.686 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.83883.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.83883.log.gz 2024-09-12T06:59:44.694 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48025.log 2024-09-12T06:59:44.695 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.admin.90079.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.90079.log.gz 2024-09-12T06:59:44.708 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44680.log 2024-09-12T06:59:44.713 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.48025.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.44723.log.gz 2024-09-12T06:59:44.716 INFO:teuthology.orchestra.run.smithi152.stderr: 92.9% -- replaced with /var/log/ceph/cluster2-client.mirror.44813.log.gz 2024-09-12T06:59:44.719 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72633.log 2024-09-12T06:59:44.721 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.44680.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.44637.log.gz 2024-09-12T06:59:44.729 INFO:teuthology.orchestra.run.smithi152.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44567.log.gz 2024-09-12T06:59:44.738 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89622.log 2024-09-12T06:59:44.738 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.72633.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.72633.log.gz 2024-09-12T06:59:44.746 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45669.log 2024-09-12T06:59:44.746 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.89622.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.89622.log.gz 2024-09-12T06:59:44.754 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73848.log 2024-09-12T06:59:44.773 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.45669.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86833.log 2024-09-12T06:59:44.773 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.73848.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.73848.log.gz 2024-09-12T06:59:44.781 INFO:teuthology.orchestra.run.smithi152.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44052.log.gz 2024-09-12T06:59:44.789 INFO:teuthology.orchestra.run.smithi152.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.39114.log.gz 2024-09-12T06:59:44.792 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46370.log 2024-09-12T06:59:44.792 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.86833.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.86833.log.gz 2024-09-12T06:59:44.797 INFO:teuthology.orchestra.run.smithi152.stderr: 95.9% -- replaced with /var/log/ceph/cluster2-client.mirror.55606.log.gz 2024-09-12T06:59:44.799 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40354.log 2024-09-12T06:59:44.801 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.46370.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76982.log 2024-09-12T06:59:44.802 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.40354.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39308.log 2024-09-12T06:59:44.807 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.76982.log: 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.76982.log.gz 2024-09-12T06:59:44.807 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71647.log 2024-09-12T06:59:44.807 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.39308.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69201.log 2024-09-12T06:59:44.809 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.71647.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.71647.log.gz 2024-09-12T06:59:44.809 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89672.log 2024-09-12T06:59:44.809 INFO:teuthology.orchestra.run.smithi152.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.39308.log.gz 2024-09-12T06:59:44.811 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.69201.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69201.log.gz 2024-09-12T06:59:44.811 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45997.log 2024-09-12T06:59:44.811 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.89672.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.89672.log.gz 2024-09-12T06:59:44.811 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82562.log 2024-09-12T06:59:44.814 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.45997.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47753.log 2024-09-12T06:59:44.818 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.82562.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.82562.log.gz 2024-09-12T06:59:44.823 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84612.log 2024-09-12T06:59:44.823 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.47753.log: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47753.log.gz 2024-09-12T06:59:44.836 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76141.log 2024-09-12T06:59:44.837 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.84612.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.84612.log.gz 2024-09-12T06:59:44.838 INFO:teuthology.orchestra.run.smithi152.stderr: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.87277.log.gz 2024-09-12T06:59:44.841 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75854.log 2024-09-12T06:59:44.842 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.76141.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45169.log.gz 2024-09-12T06:59:44.842 INFO:teuthology.orchestra.run.smithi152.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.76141.log.gz 2024-09-12T06:59:44.851 INFO:teuthology.orchestra.run.smithi152.stderr: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.71368.log.gz 2024-09-12T06:59:44.851 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35825.log 2024-09-12T06:59:44.853 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.75854.log: /var/log/ceph/cluster2-client.admin.35825.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61432.log 2024-09-12T06:59:44.853 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42658.log 2024-09-12T06:59:44.853 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.61432.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.61432.log.gz 2024-09-12T06:59:44.854 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71346.log 2024-09-12T06:59:44.858 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.42658.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42751.log 2024-09-12T06:59:44.864 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.71346.log: 91.7% 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.42658.log.gz 2024-09-12T06:59:44.864 INFO:teuthology.orchestra.run.smithi152.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.71346.log.gz 2024-09-12T06:59:44.865 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45279.log 2024-09-12T06:59:44.865 INFO:teuthology.orchestra.run.smithi152.stderr: 80.2% -- replaced with /var/log/ceph/cluster2-client.admin.35825.log.gz 2024-09-12T06:59:44.867 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.42751.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80896.log 2024-09-12T06:59:44.867 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.45279.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89340.log 2024-09-12T06:59:44.868 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.80896.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.80896.log.gz 2024-09-12T06:59:44.869 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82804.log 2024-09-12T06:59:44.869 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.89340.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.89340.log.gz 2024-09-12T06:59:44.871 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89070.log 2024-09-12T06:59:44.871 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.82804.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.82804.log.gz 2024-09-12T06:59:44.874 INFO:teuthology.orchestra.run.smithi152.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.42751.log.gz 2024-09-12T06:59:44.875 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89258.log 2024-09-12T06:59:44.881 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.89070.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45279.log.gz 2024-09-12T06:59:44.888 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38047.log 2024-09-12T06:59:44.888 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.89258.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.89258.log.gz 2024-09-12T06:59:44.898 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86812.log 2024-09-12T06:59:44.899 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.38047.log: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.87572.log.gz 2024-09-12T06:59:44.903 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86895.log 2024-09-12T06:59:44.904 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.86812.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.86812.log.gz 2024-09-12T06:59:44.904 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.38007.log 2024-09-12T06:59:44.904 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.86895.log: 83.4% -- replaced with /var/log/ceph/cluster2-client.mirror.86895.log.gz 2024-09-12T06:59:44.904 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37566.log 2024-09-12T06:59:44.904 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.1.38007.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88867.log 2024-09-12T06:59:44.904 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.37566.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85130.log 2024-09-12T06:59:44.905 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.88867.log: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.88867.log.gz 2024-09-12T06:59:44.906 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45363.log 2024-09-12T06:59:44.906 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.85130.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.85130.log.gz 2024-09-12T06:59:44.908 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80369.log 2024-09-12T06:59:44.909 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.45363.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89454.log 2024-09-12T06:59:44.910 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.80369.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.80369.log.gz 2024-09-12T06:59:44.910 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40023.log 2024-09-12T06:59:44.911 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.89454.log: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.89454.log.gz 2024-09-12T06:59:44.913 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44610.log 2024-09-12T06:59:44.913 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.admin.40023.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41113.log 2024-09-12T06:59:44.913 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.44610.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83425.log 2024-09-12T06:59:44.917 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.admin.41113.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68823.log 2024-09-12T06:59:44.918 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.83425.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.83425.log.gz 2024-09-12T06:59:44.919 INFO:teuthology.orchestra.run.smithi152.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.39969.log.gz 2024-09-12T06:59:44.920 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88117.log 2024-09-12T06:59:44.921 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.68823.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.68823.log.gz 2024-09-12T06:59:44.921 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78219.log 2024-09-12T06:59:44.923 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.88117.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.88117.log.gz 2024-09-12T06:59:44.923 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81748.log 2024-09-12T06:59:44.923 INFO:teuthology.orchestra.run.smithi152.stderr: 88.7% -- replaced with /var/log/ceph/cluster1-client.admin.40023.log.gz 2024-09-12T06:59:44.923 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.78219.log: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52754.log.gz 2024-09-12T06:59:44.926 INFO:teuthology.orchestra.run.smithi152.stderr: 68.6% -- replaced with /var/log/ceph/cluster1-client.mirror.37566.log.gz 2024-09-12T06:59:44.927 INFO:teuthology.orchestra.run.smithi152.stderr: 19.3% -- replaced with /var/log/ceph/cluster2-client.mirror.1.38007.log.gz 2024-09-12T06:59:44.931 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37863.log 2024-09-12T06:59:44.932 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.81748.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.81748.log.gz 2024-09-12T06:59:44.932 INFO:teuthology.orchestra.run.smithi152.stderr: 88.9% -- replaced with /var/log/ceph/cluster1-client.admin.41113.log.gz 2024-09-12T06:59:44.932 INFO:teuthology.orchestra.run.smithi152.stderr: 91.2% -- replaced with /var/log/ceph/cluster2-client.mirror.45669.log.gz 2024-09-12T06:59:44.935 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87422.log 2024-09-12T06:59:44.935 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.admin.37863.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37863.log.gz 2024-09-12T06:59:44.936 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70883.log 2024-09-12T06:59:44.936 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.87422.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.87422.log.gz 2024-09-12T06:59:44.936 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44504.log 2024-09-12T06:59:44.936 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.70883.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.70883.log.gz 2024-09-12T06:59:44.939 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.32936.log 2024-09-12T06:59:44.939 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.44504.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77249.log 2024-09-12T06:59:44.940 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.admin.32936.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.32936.log.gz 2024-09-12T06:59:44.940 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64874.log 2024-09-12T06:59:44.940 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.77249.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.77249.log.gz 2024-09-12T06:59:44.940 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44856.log 2024-09-12T06:59:44.940 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.64874.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.64874.log.gz 2024-09-12T06:59:44.944 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.33005.log 2024-09-12T06:59:44.944 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.44856.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86097.log 2024-09-12T06:59:44.944 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.admin.33005.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.33005.log.gz 2024-09-12T06:59:44.948 INFO:teuthology.orchestra.run.smithi152.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.45363.log.gz 2024-09-12T06:59:44.948 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37897.log 2024-09-12T06:59:44.948 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.86097.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.86097.log.gz 2024-09-12T06:59:44.963 INFO:teuthology.orchestra.run.smithi152.stderr: 95.4% -- replaced with /var/log/ceph/cluster2-client.mirror.53716.log.gz 2024-09-12T06:59:44.965 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83060.log 2024-09-12T06:59:44.965 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.37897.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80269.log 2024-09-12T06:59:44.966 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.83060.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78400.log 2024-09-12T06:59:44.967 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.80269.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.83060.log.gz 2024-09-12T06:59:44.969 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.78400.log: 88.2%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41817.log 2024-09-12T06:59:44.969 INFO:teuthology.orchestra.run.smithi152.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.78400.log.gz 2024-09-12T06:59:44.969 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81841.log 2024-09-12T06:59:44.969 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.41817.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.80269.log.gz 2024-09-12T06:59:44.972 INFO:teuthology.orchestra.run.smithi152.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.41817.log.gz 2024-09-12T06:59:44.978 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46347.log 2024-09-12T06:59:44.978 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46518.log 2024-09-12T06:59:44.979 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.81841.log: /var/log/ceph/cluster2-client.mirror.46347.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.81841.log.gz 2024-09-12T06:59:44.979 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78435.log 2024-09-12T06:59:44.981 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.46518.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73925.log 2024-09-12T06:59:44.981 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.78435.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.78435.log.gz 2024-09-12T06:59:44.982 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74303.log 2024-09-12T06:59:44.983 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.73925.log: 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.73925.log.gz 2024-09-12T06:59:44.985 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37549.log 2024-09-12T06:59:44.985 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.74303.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37643.log 2024-09-12T06:59:44.986 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.37549.log: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.68890.log.gz 2024-09-12T06:59:44.989 INFO:teuthology.orchestra.run.smithi152.stderr:gzip 94.3% -- replaced with /var/log/ceph/cluster2-client.mirror.1.38047.log.gz 2024-09-12T06:59:44.989 INFO:teuthology.orchestra.run.smithi152.stderr: -5 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.74303.log.gz 2024-09-12T06:59:44.989 INFO:teuthology.orchestra.run.smithi152.stderr: --verbose -- /var/log/ceph/cluster1-client.mirror.85490.log 2024-09-12T06:59:44.989 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.admin.37643.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37643.log.gz 2024-09-12T06:59:44.990 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47776.log 2024-09-12T06:59:44.990 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.85490.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77204.log 2024-09-12T06:59:44.990 INFO:teuthology.orchestra.run.smithi152.stderr: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.85490.log.gz 2024-09-12T06:59:44.990 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.47776.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38933.log 2024-09-12T06:59:44.990 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45867.log 2024-09-12T06:59:44.990 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.77204.log: /var/log/ceph/cluster2-client.mirror.38933.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76618.log 2024-09-12T06:59:44.992 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.45867.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37669.log 2024-09-12T06:59:44.997 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.76618.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87727.log 2024-09-12T06:59:44.997 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.admin.37669.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68170.log 2024-09-12T06:59:44.997 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.87727.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.87727.log.gz 2024-09-12T06:59:44.998 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44590.log 2024-09-12T06:59:44.998 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.68170.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.68170.log.gz 2024-09-12T06:59:44.998 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81699.log 2024-09-12T06:59:44.998 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.44590.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56649.log 2024-09-12T06:59:44.998 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.81699.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.81699.log.gz 2024-09-12T06:59:44.998 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44966.log 2024-09-12T06:59:44.998 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.56649.log: 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.56649.log.gz 2024-09-12T06:59:45.000 INFO:teuthology.orchestra.run.smithi152.stderr: 93.1% -- replaced with /var/log/ceph/cluster1-client.mirror.45009.log.gz 2024-09-12T06:59:45.000 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86662.log 2024-09-12T06:59:45.000 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.44966.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60122.log 2024-09-12T06:59:45.003 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.86662.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.38933.log.gz 2024-09-12T06:59:45.004 INFO:teuthology.orchestra.run.smithi152.stderr: 69.9% -- replaced with /var/log/ceph/cluster1-client.mirror.37549.log.gz 2024-09-12T06:59:45.007 INFO:teuthology.orchestra.run.smithi152.stderr: 80.1% -- replaced with /var/log/ceph/cluster2-client.admin.37669.log.gz 2024-09-12T06:59:45.018 INFO:teuthology.orchestra.run.smithi152.stderr: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.44610.log.gz 2024-09-12T06:59:45.019 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69520.log 2024-09-12T06:59:45.020 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.60122.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.60122.log.gz 2024-09-12T06:59:45.031 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61531.log 2024-09-12T06:59:45.031 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.69520.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.69520.log.gz 2024-09-12T06:59:45.044 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40082.log 2024-09-12T06:59:45.054 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.61531.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65351.log 2024-09-12T06:59:45.055 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.40082.log: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.44680.log.gz 2024-09-12T06:59:45.059 INFO:teuthology.orchestra.run.smithi152.stderr: 96.8% -- replaced with /var/log/ceph/cluster2-client.mirror.38659.log.gz 2024-09-12T06:59:45.061 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38772.log 2024-09-12T06:59:45.079 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.65351.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37515.log 2024-09-12T06:59:45.088 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.38772.log: 93.1% -- replaced with /var/log/ceph/cluster1-client.mirror.43484.log.gz 2024-09-12T06:59:45.095 INFO:teuthology.orchestra.run.smithi152.stderr: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.75854.log.gz 2024-09-12T06:59:45.106 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86759.log 2024-09-12T06:59:45.116 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.37515.log: 2.3% -- replaced with /var/log/ceph/cluster1-client.mirror.37515.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74644.log 2024-09-12T06:59:45.116 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-09-12T06:59:45.125 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.86759.log: 91.3%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44414.log 2024-09-12T06:59:45.131 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.74644.log: -- replaced with /var/log/ceph/cluster2-client.mirror.86759.log.gz 2024-09-12T06:59:45.138 INFO:teuthology.orchestra.run.smithi152.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.74644.log.gz 2024-09-12T06:59:45.144 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75408.log 2024-09-12T06:59:45.167 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.44414.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80036.log 2024-09-12T06:59:45.168 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.75408.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.75408.log.gz 2024-09-12T06:59:45.176 INFO:teuthology.orchestra.run.smithi152.stderr: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47776.log.gz 2024-09-12T06:59:45.183 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82613.log 2024-09-12T06:59:45.184 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.80036.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.80036.log.gz 2024-09-12T06:59:45.185 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-osd.2.log 2024-09-12T06:59:45.186 INFO:teuthology.orchestra.run.smithi152.stderr: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.76618.log.gz 2024-09-12T06:59:45.187 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.82613.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.82613.log.gz 2024-09-12T06:59:45.187 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87937.log 2024-09-12T06:59:45.189 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-osd.2.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63445.log 2024-09-12T06:59:45.189 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.87937.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84276.log 2024-09-12T06:59:45.189 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.63445.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.87937.log.gz 2024-09-12T06:59:45.191 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71724.log 2024-09-12T06:59:45.191 INFO:teuthology.orchestra.run.smithi152.stderr: 94.4%/var/log/ceph/cluster1-client.mirror.84276.log: -- replaced with /var/log/ceph/cluster1-client.mirror.38047.log.gz 2024-09-12T06:59:45.191 INFO:teuthology.orchestra.run.smithi152.stderr: 2.6% -- replaced with /var/log/ceph/cluster1-client.mirror.84276.log.gz 2024-09-12T06:59:45.191 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80725.log 2024-09-12T06:59:45.194 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.71724.log: 91.3% -- replaced with /var/log/ceph/cluster1-client.mirror.71724.log.gz 2024-09-12T06:59:45.194 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43372.log 2024-09-12T06:59:45.195 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.80725.log: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48025.log.gz 2024-09-12T06:59:45.195 INFO:teuthology.orchestra.run.smithi152.stderr: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.80725.log.gz 2024-09-12T06:59:45.196 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83538.log 2024-09-12T06:59:45.197 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.43372.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83040.log 2024-09-12T06:59:45.197 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.83538.log: 84.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46347.log.gz 2024-09-12T06:59:45.199 INFO:teuthology.orchestra.run.smithi152.stderr: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.83538.log.gz 2024-09-12T06:59:45.199 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42379.log 2024-09-12T06:59:45.200 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.83040.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.83040.log.gz 2024-09-12T06:59:45.200 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83202.log 2024-09-12T06:59:45.201 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.42379.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63346.log 2024-09-12T06:59:45.202 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.83202.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.43372.log.gz 2024-09-12T06:59:45.202 INFO:teuthology.orchestra.run.smithi152.stderr: 93.1% -- replaced with /var/log/ceph/cluster1-client.mirror.44966.log.gz 2024-09-12T06:59:45.204 INFO:teuthology.orchestra.run.smithi152.stderr: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.83202.log.gz 2024-09-12T06:59:45.209 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44434.log 2024-09-12T06:59:45.210 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.63346.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.63346.log.gz 2024-09-12T06:59:45.210 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45122.log 2024-09-12T06:59:45.211 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.44434.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61818.log 2024-09-12T06:59:45.217 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.45122.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.42379.log.gz 2024-09-12T06:59:45.220 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44348.log 2024-09-12T06:59:45.222 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.61818.log: 87.9%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85107.log 2024-09-12T06:59:45.222 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.44348.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83467.log 2024-09-12T06:59:45.222 INFO:teuthology.orchestra.run.smithi152.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.61818.log.gz 2024-09-12T06:59:45.223 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.85107.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.85107.log.gz 2024-09-12T06:59:45.226 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40242.log 2024-09-12T06:59:45.227 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.83467.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44504.log.gz 2024-09-12T06:59:45.229 INFO:teuthology.orchestra.run.smithi152.stderr: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.83467.log.gz 2024-09-12T06:59:45.240 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88139.log 2024-09-12T06:59:45.240 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78732.log 2024-09-12T06:59:45.240 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.88139.log: /var/log/ceph/cluster2-client.mirror.40242.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.88139.log.gz 2024-09-12T06:59:45.240 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77583.log 2024-09-12T06:59:45.241 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.78732.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.78732.log.gz 2024-09-12T06:59:45.241 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40625.log 2024-09-12T06:59:45.241 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.77583.log: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.77204.log.gz 2024-09-12T06:59:45.243 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46582.log 2024-09-12T06:59:45.243 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.40625.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84059.log 2024-09-12T06:59:45.244 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.46582.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85197.log 2024-09-12T06:59:45.245 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.84059.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.84059.log.gz 2024-09-12T06:59:45.251 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45079.log 2024-09-12T06:59:45.251 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.85197.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.85197.log.gz 2024-09-12T06:59:45.251 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38682.log 2024-09-12T06:59:45.259 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.45079.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79577.log 2024-09-12T06:59:45.259 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.38682.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43186.log 2024-09-12T06:59:45.262 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.79577.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.79577.log.gz 2024-09-12T06:59:45.264 INFO:teuthology.orchestra.run.smithi152.stderr: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46582.log.gz 2024-09-12T06:59:45.266 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48953.log 2024-09-12T06:59:45.268 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.43186.log: gzip -5/var/log/ceph/cluster2-client.mirror.48953.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44414.log.gz 2024-09-12T06:59:45.268 INFO:teuthology.orchestra.run.smithi152.stderr: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48953.log.gz 2024-09-12T06:59:45.269 INFO:teuthology.orchestra.run.smithi152.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.80198.log 2024-09-12T06:59:45.269 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80085.log 2024-09-12T06:59:45.271 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.80198.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.80198.log.gz 2024-09-12T06:59:45.271 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42007.log 2024-09-12T06:59:45.276 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.80085.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.80085.log.gz 2024-09-12T06:59:45.276 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37822.log 2024-09-12T06:59:45.276 INFO:teuthology.orchestra.run.smithi152.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.78219.log.gz 2024-09-12T06:59:45.276 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.42007.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84365.log 2024-09-12T06:59:45.276 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.37822.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86595.log 2024-09-12T06:59:45.276 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.84365.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78105.log 2024-09-12T06:59:45.276 INFO:teuthology.orchestra.run.smithi152.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.84365.log.gz 2024-09-12T06:59:45.276 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.86595.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76905.log 2024-09-12T06:59:45.277 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.78105.log: 83.4% -- replaced with /var/log/ceph/cluster2-client.mirror.86595.log.gz 2024-09-12T06:59:45.277 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67295.log 2024-09-12T06:59:45.283 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.76905.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.76905.log.gz 2024-09-12T06:59:45.283 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85580.log 2024-09-12T06:59:45.283 INFO:teuthology.orchestra.run.smithi152.stderr: 91.9% -- replaced with /var/log/ceph/cluster2-client.mirror.78105.log.gz 2024-09-12T06:59:45.284 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.67295.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88559.log 2024-09-12T06:59:45.284 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.85580.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45122.log.gz 2024-09-12T06:59:45.284 INFO:teuthology.orchestra.run.smithi152.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.85580.log.gz 88.1% 2024-09-12T06:59:45.284 INFO:teuthology.orchestra.run.smithi152.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.67295.log.gz 2024-09-12T06:59:45.284 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36884.log 2024-09-12T06:59:45.284 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.37953.log 2024-09-12T06:59:45.284 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.admin.36884.log: /var/log/ceph/cluster2-client.mirror.88559.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36884.log.gz 2024-09-12T06:59:45.284 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66879.log 2024-09-12T06:59:45.284 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.0.37953.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45430.log 2024-09-12T06:59:45.284 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.66879.log: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.65351.log.gz 2024-09-12T06:59:45.284 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87229.log 2024-09-12T06:59:45.287 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.45430.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61153.log 2024-09-12T06:59:45.288 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.87229.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.87229.log.gz 2024-09-12T06:59:45.289 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87769.log 2024-09-12T06:59:45.294 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.61153.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44434.log.gz 2024-09-12T06:59:45.295 INFO:teuthology.orchestra.run.smithi152.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.43186.log.gz 2024-09-12T06:59:45.296 INFO:teuthology.orchestra.run.smithi152.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.77583.log.gz 2024-09-12T06:59:45.297 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41879.log 2024-09-12T06:59:45.297 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.87769.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.87769.log.gz 2024-09-12T06:59:45.298 INFO:teuthology.orchestra.run.smithi152.stderr: 54.2% -- replaced with /var/log/ceph/cluster2-client.mirror.37822.log.gz 2024-09-12T06:59:45.301 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54111.log 2024-09-12T06:59:45.301 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.41879.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82257.log 2024-09-12T06:59:45.305 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.54111.log: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.63445.log.gz 2024-09-12T06:59:45.309 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86165.log 2024-09-12T06:59:45.315 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.82257.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.82257.log.gz 2024-09-12T06:59:45.315 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86412.log 2024-09-12T06:59:45.315 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.86165.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.86165.log.gz 2024-09-12T06:59:45.315 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47133.log 2024-09-12T06:59:45.316 INFO:teuthology.orchestra.run.smithi152.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.61531.log.gz 2024-09-12T06:59:45.317 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.86412.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.86412.log.gz 2024-09-12T06:59:45.318 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84201.log 2024-09-12T06:59:45.318 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.47133.log: 95.1% -- replaced with /var/log/ceph/cluster2-client.mirror.88559.log.gz 2024-09-12T06:59:45.319 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60819.log 2024-09-12T06:59:45.320 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.84201.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.84201.log.gz 2024-09-12T06:59:45.321 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76519.log 2024-09-12T06:59:45.323 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.60819.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.60819.log.gz 2024-09-12T06:59:45.323 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36336.log 2024-09-12T06:59:45.323 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43913.log 2024-09-12T06:59:45.323 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.admin.36336.log: /var/log/ceph/cluster2-client.mirror.76519.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36336.log.gz 2024-09-12T06:59:45.323 INFO:teuthology.orchestra.run.smithi152.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.76519.log.gz 2024-09-12T06:59:45.323 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83638.log 2024-09-12T06:59:45.325 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.43913.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50802.log 2024-09-12T06:59:45.325 INFO:teuthology.orchestra.run.smithi152.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.42007.log.gz 2024-09-12T06:59:45.325 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.83638.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.83638.log.gz 2024-09-12T06:59:45.326 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36926.log 2024-09-12T06:59:45.326 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.50802.log: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50802.log.gz 2024-09-12T06:59:45.327 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5/var/log/ceph/cluster1-client.admin.36926.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36926.log.gz 2024-09-12T06:59:45.329 INFO:teuthology.orchestra.run.smithi152.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.39992.log 2024-09-12T06:59:45.331 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83763.log 2024-09-12T06:59:45.331 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.39992.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82633.log 2024-09-12T06:59:45.331 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.83763.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44328.log 2024-09-12T06:59:45.331 INFO:teuthology.orchestra.run.smithi152.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.83763.log.gz 2024-09-12T06:59:45.341 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.82633.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45189.log 2024-09-12T06:59:45.341 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.44328.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.82633.log.gz 2024-09-12T06:59:45.341 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40717.log 2024-09-12T06:59:45.342 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.45189.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46907.log 2024-09-12T06:59:45.342 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.40717.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42158.log 2024-09-12T06:59:45.342 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.46907.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80583.log 2024-09-12T06:59:45.342 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.42158.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82947.log 2024-09-12T06:59:45.342 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.80583.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88270.log 2024-09-12T06:59:45.342 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.82947.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.82947.log.gz 2024-09-12T06:59:45.342 INFO:teuthology.orchestra.run.smithi152.stderr: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.80583.log.gz 2024-09-12T06:59:45.342 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43553.log 2024-09-12T06:59:45.342 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.88270.log: -- /var/log/ceph/cluster1-client.mirror.85715.log 2024-09-12T06:59:45.342 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.43553.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43035.log 2024-09-12T06:59:45.342 INFO:teuthology.orchestra.run.smithi152.stderr: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.88270.log.gz 2024-09-12T06:59:45.342 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.85715.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67606.log 2024-09-12T06:59:45.343 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.43035.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.85715.log.gz 2024-09-12T06:59:45.343 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50647.log 2024-09-12T06:59:45.343 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.67606.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.67606.log.gz 2024-09-12T06:59:45.343 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82328.log 2024-09-12T06:59:45.344 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.50647.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.43035.log.gz 2024-09-12T06:59:45.345 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- 84.5% /var/log/ceph/cluster2-client.mirror.64075.log 2024-09-12T06:59:45.345 INFO:teuthology.orchestra.run.smithi152.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.46907.log.gz 2024-09-12T06:59:45.347 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.82328.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.82328.log.gz 2024-09-12T06:59:45.347 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83954.log 2024-09-12T06:59:45.348 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.64075.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.64075.log.gz 2024-09-12T06:59:45.349 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83182.log 2024-09-12T06:59:45.349 INFO:teuthology.orchestra.run.smithi152.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.43913.log.gz 2024-09-12T06:59:45.353 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.83954.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.83954.log.gz 2024-09-12T06:59:45.354 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60167.log 2024-09-12T06:59:45.357 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.83182.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.83182.log.gz 2024-09-12T06:59:45.358 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75719.log 2024-09-12T06:59:45.362 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.60167.log: 91.6% -- replaced with /var/log/ceph/cluster1-client.mirror.60167.log.gz 2024-09-12T06:59:45.364 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61509.log 2024-09-12T06:59:45.365 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.75719.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.75719.log.gz 2024-09-12T06:59:45.365 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64488.log 2024-09-12T06:59:45.367 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.61509.log: 91.7% -- replaced with /var/log/ceph/cluster1-client.mirror.61509.log.gz 2024-09-12T06:59:45.369 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69555.log 2024-09-12T06:59:45.376 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.64488.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64488.log.gz 2024-09-12T06:59:45.376 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37136.log 2024-09-12T06:59:45.377 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.69555.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.69555.log.gz 2024-09-12T06:59:45.377 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43221.log 2024-09-12T06:59:45.377 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.admin.37136.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37136.log.gz 2024-09-12T06:59:45.378 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86854.log 2024-09-12T06:59:45.382 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.43221.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51135.log 2024-09-12T06:59:45.387 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.86854.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.86854.log.gz 2024-09-12T06:59:45.387 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81890.log 2024-09-12T06:59:45.392 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.51135.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.41879.log.gz 2024-09-12T06:59:45.393 INFO:teuthology.orchestra.run.smithi152.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.42158.log.gz 2024-09-12T06:59:45.395 INFO:teuthology.orchestra.run.smithi152.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.43221.log.gz 2024-09-12T06:59:45.397 INFO:teuthology.orchestra.run.smithi152.stderr: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.44348.log.gz 2024-09-12T06:59:45.399 INFO:teuthology.orchestra.run.smithi152.stderr: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.45079.log.gz 2024-09-12T06:59:45.402 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42565.log 2024-09-12T06:59:45.403 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.81890.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.81890.log.gz 2024-09-12T06:59:45.413 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45233.log 2024-09-12T06:59:45.416 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.42565.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89299.log 2024-09-12T06:59:45.419 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.45233.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72207.log 2024-09-12T06:59:45.419 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.89299.log: 83.3% -- replaced with /var/log/ceph/cluster2-client.mirror.89299.log.gz 2024-09-12T06:59:45.419 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85062.log 2024-09-12T06:59:45.422 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.72207.log: 91.6% -- replaced with /var/log/ceph/cluster1-client.mirror.72207.log.gz 2024-09-12T06:59:45.422 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40309.log 2024-09-12T06:59:45.422 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.85062.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.85062.log.gz 2024-09-12T06:59:45.425 INFO:teuthology.orchestra.run.smithi152.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.42565.log.gz 2024-09-12T06:59:45.425 INFO:teuthology.orchestra.run.smithi152.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45233.log.gz 2024-09-12T06:59:45.426 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39661.log 2024-09-12T06:59:45.427 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.40309.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45518.log 2024-09-12T06:59:45.429 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.admin.39661.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41172.log 2024-09-12T06:59:45.446 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.45518.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44481.log 2024-09-12T06:59:45.455 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.41172.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72033.log 2024-09-12T06:59:45.458 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.44481.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44328.log.gz 2024-09-12T06:59:45.461 INFO:teuthology.orchestra.run.smithi152.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.89070.log.gz 2024-09-12T06:59:45.462 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84432.log 2024-09-12T06:59:45.463 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.72033.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.72033.log.gz 2024-09-12T06:59:45.470 INFO:teuthology.orchestra.run.smithi152.stderr: 88.6% -- replaced with /var/log/ceph/cluster1-client.admin.39661.log.gz 2024-09-12T06:59:45.478 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88300.log 2024-09-12T06:59:45.478 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.84432.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.84432.log.gz 2024-09-12T06:59:45.482 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77635.log 2024-09-12T06:59:45.482 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.88300.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.88300.log.gz 2024-09-12T06:59:45.482 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36002.log 2024-09-12T06:59:45.482 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.77635.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.77635.log.gz 2024-09-12T06:59:45.483 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87789.log 2024-09-12T06:59:45.485 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.admin.36002.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36002.log.gz 2024-09-12T06:59:45.485 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40694.log 2024-09-12T06:59:45.490 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.87789.log: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.87789.log.gz 2024-09-12T06:59:45.490 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43093.log 2024-09-12T06:59:45.491 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.40694.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80107.log 2024-09-12T06:59:45.493 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.43093.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41655.log 2024-09-12T06:59:45.494 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.80107.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.80107.log.gz 2024-09-12T06:59:45.494 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40412.log 2024-09-12T06:59:45.497 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.admin.41655.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.45189.log.gz 2024-09-12T06:59:45.497 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39004.log 2024-09-12T06:59:45.498 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.admin.40412.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87851.log 2024-09-12T06:59:45.501 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.39004.log: gzip -5 --verbose -- /var/log/ceph/cluster2.audit.log 2024-09-12T06:59:45.505 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.87851.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.87851.log.gz 2024-09-12T06:59:45.505 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89549.log 2024-09-12T06:59:45.507 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2.audit.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89528.log 2024-09-12T06:59:45.507 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.89549.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.89549.log.gz 2024-09-12T06:59:45.511 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66016.log 2024-09-12T06:59:45.512 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.89528.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.89528.log.gz 2024-09-12T06:59:45.512 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46053.log 2024-09-12T06:59:45.514 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.66016.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.66016.log.gz 2024-09-12T06:59:45.514 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52731.log 2024-09-12T06:59:45.514 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.46053.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38002.log 2024-09-12T06:59:45.515 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.52731.log: 84.4% -- replaced with /var/log/ceph/cluster2-client.mirror.52731.log.gz 2024-09-12T06:59:45.515 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66966.log 2024-09-12T06:59:45.518 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.38002.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71746.log 2024-09-12T06:59:45.518 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.66966.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.66966.log.gz 2024-09-12T06:59:45.518 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44903.log 2024-09-12T06:59:45.537 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.71746.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85850.log 2024-09-12T06:59:45.543 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.44903.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44481.log.gz 2024-09-12T06:59:45.544 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66115.log 2024-09-12T06:59:45.544 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.85850.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.85850.log.gz 2024-09-12T06:59:45.551 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42716.log 2024-09-12T06:59:45.556 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.66115.log: 96.0% -- replaced with /var/log/ceph/cluster2-client.mirror.66115.log.gz 2024-09-12T06:59:45.558 INFO:teuthology.orchestra.run.smithi152.stderr: 95.9% -- replaced with /var/log/ceph/cluster2-client.mirror.86662.log.gz 2024-09-12T06:59:45.562 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40513.log 2024-09-12T06:59:45.563 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.42716.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66657.log 2024-09-12T06:59:45.563 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.40513.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64545.log 2024-09-12T06:59:45.565 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.66657.log: 91.7% -- replaced with /var/log/ceph/cluster1-client.mirror.66657.log.gz 2024-09-12T06:59:45.565 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35876.log 2024-09-12T06:59:45.569 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.64545.log: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64545.log.gz 2024-09-12T06:59:45.570 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87305.log 2024-09-12T06:59:45.574 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.admin.35876.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35876.log.gz 2024-09-12T06:59:45.574 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79191.log 2024-09-12T06:59:45.574 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.87305.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.87305.log.gz 2024-09-12T06:59:45.575 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41782.log 2024-09-12T06:59:45.575 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.79191.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.79191.log.gz 2024-09-12T06:59:45.576 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59381.log 2024-09-12T06:59:45.584 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.41782.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82969.log 2024-09-12T06:59:45.584 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.59381.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.59381.log.gz 2024-09-12T06:59:45.584 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46830.log 2024-09-12T06:59:45.584 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.82969.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.82969.log.gz 2024-09-12T06:59:45.588 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71679.log 2024-09-12T06:59:45.588 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.46830.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45497.log 2024-09-12T06:59:45.589 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.71679.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.71679.log.gz 2024-09-12T06:59:45.590 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60545.log 2024-09-12T06:59:45.590 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.45497.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36758.log 2024-09-12T06:59:45.591 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.60545.log: 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.60545.log.gz 2024-09-12T06:59:45.593 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45539.log 2024-09-12T06:59:45.593 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.admin.36758.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36758.log.gz 2024-09-12T06:59:45.594 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42100.log 2024-09-12T06:59:45.596 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.45539.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64565.log 2024-09-12T06:59:45.596 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.42100.log: 19.3% -- replaced with /var/log/ceph/cluster2-client.mirror.0.37953.log.gz 2024-09-12T06:59:45.597 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.90247.log 2024-09-12T06:59:45.598 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.64565.log: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.64565.log.gz 2024-09-12T06:59:45.599 INFO:teuthology.orchestra.run.smithi152.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.43093.log.gz 2024-09-12T06:59:45.600 INFO:teuthology.orchestra.run.smithi152.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.39004.log.gz 2024-09-12T06:59:45.602 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72678.log 2024-09-12T06:59:45.602 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.admin.90247.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.90247.log.gz 2024-09-12T06:59:45.603 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.32797.log 2024-09-12T06:59:45.606 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.72678.log: 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.72678.log.gz 2024-09-12T06:59:45.606 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82733.log 2024-09-12T06:59:45.606 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.admin.32797.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.32797.log.gz 2024-09-12T06:59:45.607 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82073.log 2024-09-12T06:59:45.608 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.82733.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.82733.log.gz 2024-09-12T06:59:45.608 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86232.log 2024-09-12T06:59:45.610 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.82073.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.82073.log.gz 2024-09-12T06:59:45.610 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78125.log 2024-09-12T06:59:45.611 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.86232.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.86232.log.gz 2024-09-12T06:59:45.611 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64209.log 2024-09-12T06:59:45.612 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.78125.log: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50647.log.gz 2024-09-12T06:59:45.614 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.64209.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66093.log 2024-09-12T06:59:45.615 INFO:teuthology.orchestra.run.smithi152.stderr: 91.7% -- replaced with /var/log/ceph/cluster1-client.mirror.78125.log.gz 2024-09-12T06:59:45.615 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43836.log 2024-09-12T06:59:45.616 INFO:teuthology.orchestra.run.smithi152.stderr: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.71746.log.gz 2024-09-12T06:59:45.617 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.66093.log: 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66093.log.gz 2024-09-12T06:59:45.617 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37481.log 2024-09-12T06:59:45.618 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.43836.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88962.log 2024-09-12T06:59:45.619 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.admin.37481.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86142.log 2024-09-12T06:59:45.620 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.88962.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.88962.log.gz 2024-09-12T06:59:45.622 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77947.log 2024-09-12T06:59:45.623 INFO:teuthology.orchestra.run.smithi152.stderr:gzip/var/log/ceph/cluster1-client.mirror.86142.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80654.log 2024-09-12T06:59:45.623 INFO:teuthology.orchestra.run.smithi152.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.86142.log.gz 2024-09-12T06:59:45.623 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.77947.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75433.log 2024-09-12T06:59:45.626 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.80654.log: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.77947.log.gz 2024-09-12T06:59:45.626 INFO:teuthology.orchestra.run.smithi152.stderr: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.80654.log.gz 2024-09-12T06:59:45.626 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85422.log 2024-09-12T06:59:45.626 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62760.log 2024-09-12T06:59:45.626 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.85422.log: /var/log/ceph/cluster2-client.mirror.75433.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.85422.log.gz 2024-09-12T06:59:45.626 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85692.log 2024-09-12T06:59:45.626 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.62760.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.62760.log.gz 2024-09-12T06:59:45.630 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.85692.log: 89.1% -- replaced with /var/log/ceph/cluster1-client.admin.41655.log.gz 2024-09-12T06:59:45.631 INFO:teuthology.orchestra.run.smithi152.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.41782.log.gz 2024-09-12T06:59:45.631 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose 88.7% -- -- replaced with /var/log/ceph/cluster2-client.mirror.42100.log.gz 2024-09-12T06:59:45.631 INFO:teuthology.orchestra.run.smithi152.stderr: /var/log/ceph/cluster2-client.mirror.38840.log 2024-09-12T06:59:45.631 INFO:teuthology.orchestra.run.smithi152.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.85692.log.gz 2024-09-12T06:59:45.631 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.90373.log 2024-09-12T06:59:45.631 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.38840.log: 91.8%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.37888.log 2024-09-12T06:59:45.631 INFO:teuthology.orchestra.run.smithi152.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.75433.log.gz 2024-09-12T06:59:45.631 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.admin.90373.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.90373.log.gz 2024-09-12T06:59:45.631 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74612.log 2024-09-12T06:59:45.631 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78890.log 2024-09-12T06:59:45.631 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.74612.log: /var/log/ceph/cluster1-client.mirror.0.37888.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.74612.log.gz 2024-09-12T06:59:45.631 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47273.log 2024-09-12T06:59:45.637 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.78890.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.78890.log.gz 2024-09-12T06:59:45.637 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56897.log 2024-09-12T06:59:45.637 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37703.log 2024-09-12T06:59:45.637 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.56897.log: /var/log/ceph/cluster2-client.mirror.47273.log: 1.3% -- replaced with /var/log/ceph/cluster2-client.mirror.56897.log.gz 2024-09-12T06:59:45.637 INFO:teuthology.orchestra.run.smithi152.stderr: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47273.log.gz 2024-09-12T06:59:45.637 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35653.log 2024-09-12T06:59:45.637 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.admin.37703.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72429.log 2024-09-12T06:59:45.637 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36590.log 2024-09-12T06:59:45.638 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.72429.log: /var/log/ceph/cluster2-client.admin.35653.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35653.log.gz 2024-09-12T06:59:45.638 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38744.log 2024-09-12T06:59:45.638 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.admin.36590.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36590.log.gz 2024-09-12T06:59:45.638 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42530.log 2024-09-12T06:59:45.640 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.admin.38744.log: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51135.log.gz 2024-09-12T06:59:45.641 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.37941.log 2024-09-12T06:59:45.641 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.42530.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37178.log 2024-09-12T06:59:45.641 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.1.37941.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60189.log 2024-09-12T06:59:45.641 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.admin.37178.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37178.log.gz 2024-09-12T06:59:45.642 INFO:teuthology.orchestra.run.smithi152.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.42530.log.gz 2024-09-12T06:59:45.642 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.42716.log.gz 2024-09-12T06:59:45.642 INFO:teuthology.orchestra.run.smithi152.stderr: /var/log/ceph/cluster2-client.mirror.68205.log 2024-09-12T06:59:45.643 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.60189.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.38744.log.gz 2024-09-12T06:59:45.645 INFO:teuthology.orchestra.run.smithi152.stderr: 75.0% -- replaced with /var/log/ceph/cluster1-client.admin.37481.log.gz 2024-09-12T06:59:45.645 INFO:teuthology.orchestra.run.smithi152.stderr: 96.8% -- replaced with /var/log/ceph/cluster2-client.mirror.72429.log.gz 2024-09-12T06:59:45.645 INFO:teuthology.orchestra.run.smithi152.stderr:gzip 84.6% -5 -- replaced with /var/log/ceph/cluster2-client.mirror.46053.log.gz 2024-09-12T06:59:45.645 INFO:teuthology.orchestra.run.smithi152.stderr: --verbose -- /var/log/ceph/cluster2-client.admin.37686.log 2024-09-12T06:59:45.645 INFO:teuthology.orchestra.run.smithi152.stderr: 74.5% -- replaced with /var/log/ceph/cluster2-client.admin.37703.log.gz 2024-09-12T06:59:45.645 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.68205.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.68205.log.gz 2024-09-12T06:59:45.660 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85467.log 2024-09-12T06:59:45.660 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.admin.37686.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43128.log 2024-09-12T06:59:45.660 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 19.3% --verbose -- replaced with /var/log/ceph/cluster1-client.mirror.0.37888.log.gz 2024-09-12T06:59:45.660 INFO:teuthology.orchestra.run.smithi152.stderr: -- /var/log/ceph/cluster2-client.mirror.41370.log 2024-09-12T06:59:45.660 INFO:teuthology.orchestra.run.smithi152.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.40513.log.gz 2024-09-12T06:59:45.661 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.43128.log: /var/log/ceph/cluster1-client.mirror.85467.log: gzip -5 --verbose -- 86.9% /var/log/ceph/cluster1-client.mirror.64906.log 2024-09-12T06:59:45.661 INFO:teuthology.orchestra.run.smithi152.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.85467.log.gz 2024-09-12T06:59:45.661 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.41370.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85512.log 2024-09-12T06:59:45.661 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.64906.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70103.log 2024-09-12T06:59:45.661 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.85512.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.64906.log.gz 2024-09-12T06:59:45.661 INFO:teuthology.orchestra.run.smithi152.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.85512.log.gz 2024-09-12T06:59:45.661 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87148.log 2024-09-12T06:59:45.661 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.70103.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44391.log 2024-09-12T06:59:45.661 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.87148.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.87148.log.gz 2024-09-12T06:59:45.661 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89697.log 2024-09-12T06:59:45.661 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.44391.log: 91.8% -- replaced with /var/log/ceph/cluster1-client.mirror.70103.log.gz 2024-09-12T06:59:45.661 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43778.log 2024-09-12T06:59:45.661 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.89697.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.89697.log.gz 2024-09-12T06:59:45.661 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43531.log 2024-09-12T06:59:45.662 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.43778.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73049.log 2024-09-12T06:59:45.662 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.43531.log: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47133.log.gz 2024-09-12T06:59:45.662 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44138.log 2024-09-12T06:59:45.662 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.73049.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.89767.log 2024-09-12T06:59:45.662 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.44138.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.73049.log.gz 2024-09-12T06:59:45.662 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88425.log 2024-09-12T06:59:45.662 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45212.log 2024-09-12T06:59:45.662 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.admin.89767.log: /var/log/ceph/cluster2-client.mirror.88425.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83589.log 2024-09-12T06:59:45.662 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.45212.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.89767.log.gz 2024-09-12T06:59:45.662 INFO:teuthology.orchestra.run.smithi152.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.88425.log.gz 2024-09-12T06:59:45.662 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84927.log 2024-09-12T06:59:45.662 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84657.log 2024-09-12T06:59:45.662 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.84927.log: /var/log/ceph/cluster2-client.mirror.83589.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.84927.log.gz 2024-09-12T06:59:45.662 INFO:teuthology.orchestra.run.smithi152.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.83589.log.gz 2024-09-12T06:59:45.663 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77541.log 2024-09-12T06:59:45.663 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84255.log 2024-09-12T06:59:45.663 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.77541.log: /var/log/ceph/cluster1-client.mirror.84657.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.84657.log.gz 2024-09-12T06:59:45.663 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67320.log 2024-09-12T06:59:45.663 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.84255.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45974.log 2024-09-12T06:59:45.663 INFO:teuthology.orchestra.run.smithi152.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.84255.log.gz 2024-09-12T06:59:45.663 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81557.log 2024-09-12T06:59:45.663 INFO:teuthology.orchestra.run.smithi152.stderr: 91.9% 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.54111.log.gz 2024-09-12T06:59:45.663 INFO:teuthology.orchestra.run.smithi152.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.77541.log.gz 2024-09-12T06:59:45.663 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.67320.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37532.log 2024-09-12T06:59:45.663 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.81557.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79634.log 2024-09-12T06:59:45.663 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.37532.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.81557.log.gz 2024-09-12T06:59:45.678 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.45974.log: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.67320.log.gz 2024-09-12T06:59:45.678 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87992.log 2024-09-12T06:59:45.678 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47658.log 2024-09-12T06:59:45.678 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.79634.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89647.log 2024-09-12T06:59:45.678 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.87992.log: /var/log/ceph/cluster2-client.mirror.47658.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75786.log 2024-09-12T06:59:45.678 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.89647.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.87992.log.gz 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.89647.log.gz 2024-09-12T06:59:45.678 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-09-12T06:59:45.679 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85400.log 2024-09-12T06:59:45.679 INFO:teuthology.orchestra.run.smithi152.stderr: 92.1% -- replaced with /var/log/ceph/cluster2-client.mirror.79634.log.gz 2024-09-12T06:59:45.679 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.75786.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73905.log 2024-09-12T06:59:45.679 INFO:teuthology.orchestra.run.smithi152.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.75786.log.gz 2024-09-12T06:59:45.679 INFO:teuthology.orchestra.run.smithi152.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.38840.log.gz/var/log/ceph/cluster1-client.mirror.85400.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39165.log 2024-09-12T06:59:45.679 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.73905.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.85400.log.gz 2024-09-12T06:59:45.679 INFO:teuthology.orchestra.run.smithi152.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.43778.log.gz 2024-09-12T06:59:45.679 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40054.log 2024-09-12T06:59:45.679 INFO:teuthology.orchestra.run.smithi152.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.43836.log.gz 2024-09-12T06:59:45.679 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-09-12T06:59:45.679 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.admin.39165.log: gzip -5 --verbose -- 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.73905.log.gz 2024-09-12T06:59:45.679 INFO:teuthology.orchestra.run.smithi152.stderr: /var/log/ceph/cluster2-client.mirror.62725.log 2024-09-12T06:59:45.679 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84130.log 2024-09-12T06:59:45.680 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.62725.log: /var/log/ceph/cluster1-client.admin.40054.log: gzip 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.62725.log.gz 2024-09-12T06:59:45.680 INFO:teuthology.orchestra.run.smithi152.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77284.log 2024-09-12T06:59:45.680 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.84130.log: /var/log/ceph/cluster2-client.mirror.77284.log: 88.5% -- replaced with /var/log/ceph/cluster1-client.admin.40412.log.gz 2024-09-12T06:59:45.680 INFO:teuthology.orchestra.run.smithi152.stderr: 87.9% 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.77284.log.gz 2024-09-12T06:59:45.680 INFO:teuthology.orchestra.run.smithi152.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.41370.log.gz 2024-09-12T06:59:45.680 INFO:teuthology.orchestra.run.smithi152.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.84130.log.gz 2024-09-12T06:59:45.680 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65284.log 2024-09-12T06:59:45.680 INFO:teuthology.orchestra.run.smithi152.stderr: 88.4%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40911.log 2024-09-12T06:59:45.680 INFO:teuthology.orchestra.run.smithi152.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.40054.log.gz 2024-09-12T06:59:45.680 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.65284.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.65284.log.gz 2024-09-12T06:59:45.680 INFO:teuthology.orchestra.run.smithi152.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.43128.log.gz 2024-09-12T06:59:45.680 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45753.log 2024-09-12T06:59:45.680 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.40911.log: -- /var/log/ceph/cluster2-client.mirror.72187.log 2024-09-12T06:59:45.680 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.45753.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87524.log 2024-09-12T06:59:45.681 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.72187.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89722.log 2024-09-12T06:59:45.681 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.87524.log: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.66879.log.gz 2024-09-12T06:59:45.681 INFO:teuthology.orchestra.run.smithi152.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.87524.log.gz 2024-09-12T06:59:45.681 INFO:teuthology.orchestra.run.smithi152.stderr: 84.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45974.log.gz 2024-09-12T06:59:45.681 INFO:teuthology.orchestra.run.smithi152.stderr: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.72187.log.gz 2024-09-12T06:59:45.681 INFO:teuthology.orchestra.run.smithi152.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.44138.log.gz 2024-09-12T06:59:45.681 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68504.log 2024-09-12T06:59:45.681 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.89722.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58139.log 2024-09-12T06:59:45.681 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.89722.log.gz 2024-09-12T06:59:45.681 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.68504.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69991.log 2024-09-12T06:59:45.681 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.58139.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83305.log 2024-09-12T06:59:45.681 INFO:teuthology.orchestra.run.smithi152.stderr: 2.6% -- replaced with /var/log/ceph/cluster2-client.mirror.58139.log.gz 2024-09-12T06:59:45.681 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.69991.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.69991.log.gz 2024-09-12T06:59:45.681 INFO:teuthology.orchestra.run.smithi152.stderr: 89.3% 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.45753.log.gz 3.5% 2024-09-12T06:59:45.681 INFO:teuthology.orchestra.run.smithi152.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.37532.log.gz 2024-09-12T06:59:45.682 INFO:teuthology.orchestra.run.smithi152.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.45212.log.gz 2024-09-12T06:59:45.690 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40652.log 2024-09-12T06:59:45.690 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.83305.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75089.log 2024-09-12T06:59:45.690 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.40652.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.83305.log.gz 2024-09-12T06:59:45.690 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66580.log 2024-09-12T06:59:45.691 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose --/var/log/ceph/cluster2-client.mirror.75089.log: /var/log/ceph/cluster1-client.mirror.86007.log 2024-09-12T06:59:45.691 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.66580.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.66580.log.gz 2024-09-12T06:59:45.691 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.86007.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.86007.log.gz 2024-09-12T06:59:45.691 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44743.log 2024-09-12T06:59:45.691 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89569.log 2024-09-12T06:59:45.691 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.44743.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86255.log 2024-09-12T06:59:45.691 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46930.log 2024-09-12T06:59:45.691 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.89569.log: /var/log/ceph/cluster1-client.mirror.86255.log: 86.5% -- replaced with /var/log/ceph/cluster1-client.mirror.86255.log.gz 2024-09-12T06:59:45.691 INFO:teuthology.orchestra.run.smithi152.stderr: 91.7% -- replaced with /var/log/ceph/cluster2-client.mirror.89569.log.gz 2024-09-12T06:59:45.691 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88626.log 2024-09-12T06:59:45.691 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.46930.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75341.log 2024-09-12T06:59:45.691 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45472.log 2024-09-12T06:59:45.691 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.75341.log: /var/log/ceph/cluster1-client.mirror.88626.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.88626.log.gz 2024-09-12T06:59:45.691 INFO:teuthology.orchestra.run.smithi152.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.75341.log.gz 2024-09-12T06:59:45.692 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68868.log 2024-09-12T06:59:45.692 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.45472.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88218.log 2024-09-12T06:59:45.692 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80847.log 2024-09-12T06:59:45.692 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.88218.log: /var/log/ceph/cluster1-client.mirror.68868.log: 91.7% -- replaced with /var/log/ceph/cluster1-client.mirror.68868.log.gz 2024-09-12T06:59:45.694 INFO:teuthology.orchestra.run.smithi152.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.88218.log.gz 2024-09-12T06:59:45.694 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36674.log 2024-09-12T06:59:45.694 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.80847.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83692.log 2024-09-12T06:59:45.694 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.admin.36674.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36674.log.gz 2024-09-12T06:59:45.694 INFO:teuthology.orchestra.run.smithi152.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.80847.log.gz 2024-09-12T06:59:45.697 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74990.log 2024-09-12T06:59:45.697 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.83692.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54498.log 2024-09-12T06:59:45.697 INFO:teuthology.orchestra.run.smithi152.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.83692.log.gz/var/log/ceph/cluster2-client.mirror.74990.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.74990.log.gz 2024-09-12T06:59:45.697 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-09-12T06:59:45.701 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64167.log 2024-09-12T06:59:45.701 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.54498.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82044.log 2024-09-12T06:59:45.701 INFO:teuthology.orchestra.run.smithi152.stderr: 84.4% -- replaced with /var/log/ceph/cluster2-client.mirror.54498.log.gz 2024-09-12T06:59:45.701 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43576.log 2024-09-12T06:59:45.702 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.82044.log: /var/log/ceph/cluster2-client.mirror.64167.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.82044.log.gz 2024-09-12T06:59:45.703 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.43576.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66073.log 2024-09-12T06:59:45.704 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85647.log 2024-09-12T06:59:45.706 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.66073.log: 91.5% -- replaced with /var/log/ceph/cluster2-client.mirror.64167.log.gz 2024-09-12T06:59:45.706 INFO:teuthology.orchestra.run.smithi152.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.45430.log.gz 2024-09-12T06:59:45.707 INFO:teuthology.orchestra.run.smithi152.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.45497.log.gz 2024-09-12T06:59:45.707 INFO:teuthology.orchestra.run.smithi152.stderr: 92.1% -- replaced with /var/log/ceph/cluster2-client.mirror.66073.log.gz 2024-09-12T06:59:45.707 INFO:teuthology.orchestra.run.smithi152.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.45539.log.gz 2024-09-12T06:59:45.707 INFO:teuthology.orchestra.run.smithi152.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.45518.log.gz 2024-09-12T06:59:45.710 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46292.log 2024-09-12T06:59:45.711 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.85647.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88447.log 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.85647.log.gz 2024-09-12T06:59:45.711 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-09-12T06:59:45.711 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41551.log 2024-09-12T06:59:45.711 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.88447.log: /var/log/ceph/cluster2-client.mirror.46292.log: gzip -5 --verbose 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.88447.log.gz 2024-09-12T06:59:45.711 INFO:teuthology.orchestra.run.smithi152.stderr: -- /var/log/ceph/cluster1-client.mirror.85827.log 2024-09-12T06:59:45.715 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.41551.log: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.61153.log.gz 2024-09-12T06:59:45.715 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87546.log 2024-09-12T06:59:45.715 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose/var/log/ceph/cluster1-client.mirror.85827.log: -- /var/log/ceph/cluster2-client.mirror.61917.log 2024-09-12T06:59:45.715 INFO:teuthology.orchestra.run.smithi152.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.85827.log.gz 2024-09-12T06:59:45.715 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.87546.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87189.log 2024-09-12T06:59:45.715 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.61917.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.87546.log.gz 2024-09-12T06:59:45.715 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50325.log 2024-09-12T06:59:45.715 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37839.log 2024-09-12T06:59:45.718 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.50325.log: /var/log/ceph/cluster2-client.mirror.87189.log: 84.4% -- replaced with /var/log/ceph/cluster2-client.mirror.50325.log.gz 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.87189.log.gz 2024-09-12T06:59:45.718 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75047.log 2024-09-12T06:59:45.718 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.admin.37839.log: 0.0% 2024-09-12T06:59:45.718 INFO:teuthology.orchestra.run.smithi152.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.37839.log.gz 2024-09-12T06:59:45.718 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37583.log 2024-09-12T06:59:45.720 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.75047.log: 92.1% -- replaced with /var/log/ceph/cluster2-client.mirror.75047.log.gz 2024-09-12T06:59:45.726 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.37583.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59811.log 2024-09-12T06:59:45.726 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63689.log 2024-09-12T06:59:45.726 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46269.log 2024-09-12T06:59:45.726 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.63689.log: /var/log/ceph/cluster2-client.mirror.59811.log: 96.8% -- replaced with /var/log/ceph/cluster2-client.mirror.61917.log.gz 2024-09-12T06:59:45.726 INFO:teuthology.orchestra.run.smithi152.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.63689.log.gz 2024-09-12T06:59:45.726 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89429.log 2024-09-12T06:59:45.726 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.46269.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45720.log 2024-09-12T06:59:45.726 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose/var/log/ceph/cluster1-client.mirror.89429.log: -- /var/log/ceph/cluster2-client.mirror.51299.log 2024-09-12T06:59:45.727 INFO:teuthology.orchestra.run.smithi152.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.89429.log.gz 2024-09-12T06:59:45.727 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.45720.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62547.log 2024-09-12T06:59:45.727 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.51299.log: 84.3% -- replaced with /var/log/ceph/cluster2-client.mirror.51299.log.gz 2024-09-12T06:59:45.727 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71998.log 2024-09-12T06:59:45.727 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.62547.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.62547.log.gz 2024-09-12T06:59:45.731 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.71998.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.71998.log.gz 2024-09-12T06:59:45.731 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36210.log 2024-09-12T06:59:45.731 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83396.log 2024-09-12T06:59:45.731 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.admin.36210.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36210.log.gz 2024-09-12T06:59:45.731 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84815.log 2024-09-12T06:59:45.731 INFO:teuthology.orchestra.run.smithi152.stderr: 88.6%/var/log/ceph/cluster2-client.mirror.83396.log: -- replaced with /var/log/ceph/cluster1-client.admin.39165.log.gz 2024-09-12T06:59:45.731 INFO:teuthology.orchestra.run.smithi152.stderr: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46269.log.gz 2024-09-12T06:59:45.731 INFO:teuthology.orchestra.run.smithi152.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.40911.log.gz 2024-09-12T06:59:45.731 INFO:teuthology.orchestra.run.smithi152.stderr: 53.8% -- replaced with /var/log/ceph/cluster1-client.mirror.37583.log.gz 2024-09-12T06:59:45.732 INFO:teuthology.orchestra.run.smithi152.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.45472.log.gz 2024-09-12T06:59:45.732 INFO:teuthology.orchestra.run.smithi152.stderr: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.83396.log.gz 2024-09-12T06:59:45.739 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54747.log 2024-09-12T06:59:45.739 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.84815.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.84815.log.gz 2024-09-12T06:59:45.739 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.90415.log 2024-09-12T06:59:45.740 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.54747.log: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54747.log.gz 2024-09-12T06:59:45.740 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38968.log 2024-09-12T06:59:45.740 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60931.log 2024-09-12T06:59:45.740 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.38968.log: /var/log/ceph/cluster2-client.admin.90415.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.90415.log.gz 2024-09-12T06:59:45.740 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85085.log 2024-09-12T06:59:45.740 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.60931.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47635.log 2024-09-12T06:59:45.740 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.85085.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.85085.log.gz 2024-09-12T06:59:45.740 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87104.log 2024-09-12T06:59:45.740 INFO:teuthology.orchestra.run.smithi152.stderr: 91.8% -- replaced with /var/log/ceph/cluster1-client.mirror.60931.log.gz 2024-09-12T06:59:45.740 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62659.log 2024-09-12T06:59:45.740 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.47635.log: /var/log/ceph/cluster1-client.mirror.87104.log: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47635.log.gz 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.87104.log.gz 2024-09-12T06:59:45.740 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-09-12T06:59:45.740 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50178.log 2024-09-12T06:59:45.741 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.62659.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37617.log 2024-09-12T06:59:45.745 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.50178.log: 91.6% -- replaced with /var/log/ceph/cluster1-client.mirror.62659.log.gz 2024-09-12T06:59:45.745 INFO:teuthology.orchestra.run.smithi152.stderr: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.64209.log.gz 2024-09-12T06:59:45.745 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86480.log 2024-09-12T06:59:45.745 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.admin.37617.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37617.log.gz 2024-09-12T06:59:45.745 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66202.log 2024-09-12T06:59:45.745 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.86480.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53334.log 2024-09-12T06:59:45.745 INFO:teuthology.orchestra.run.smithi152.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.86480.log.gz 2024-09-12T06:59:45.745 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.66202.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64110.log 2024-09-12T06:59:45.745 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.53334.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.66202.log.gz 2024-09-12T06:59:45.745 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44989.log 2024-09-12T06:59:45.745 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.64110.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64110.log.gz 2024-09-12T06:59:45.745 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87210.log 2024-09-12T06:59:45.746 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.44989.log: gzip -5 --verbose -- /var/log/ceph/cluster2-osd.0.log 2024-09-12T06:59:45.746 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.87210.log: 83.3% -- replaced with /var/log/ceph/cluster2-client.mirror.87210.log.gz 2024-09-12T06:59:45.750 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84792.log 2024-09-12T06:59:45.753 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-osd.0.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45409.log 2024-09-12T06:59:45.753 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.84792.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.84792.log.gz 2024-09-12T06:59:45.753 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82989.log 2024-09-12T06:59:45.754 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.45409.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63311.log 2024-09-12T06:59:45.756 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.82989.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.82989.log.gz 2024-09-12T06:59:45.756 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71326.log 2024-09-12T06:59:45.760 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.63311.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.63311.log.gz 2024-09-12T06:59:45.764 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73519.log 2024-09-12T06:59:45.769 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.71326.log: 91.8% -- replaced with /var/log/ceph/cluster2-client.mirror.71326.log.gz 2024-09-12T06:59:45.773 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72700.log 2024-09-12T06:59:45.774 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.73519.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.38968.log.gz 2024-09-12T06:59:45.776 INFO:teuthology.orchestra.run.smithi152.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.73519.log.gz 2024-09-12T06:59:45.776 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86435.log 2024-09-12T06:59:45.785 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.72700.log: 80.1% -- replaced with /var/log/ceph/cluster2-client.admin.37686.log.gz 2024-09-12T06:59:45.786 INFO:teuthology.orchestra.run.smithi152.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.59811.log.gz 2024-09-12T06:59:45.787 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40185.log 2024-09-12T06:59:45.789 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.86435.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.86435.log.gz 2024-09-12T06:59:45.789 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45844.log 2024-09-12T06:59:45.789 INFO:teuthology.orchestra.run.smithi152.stderr: 96.6% -- replaced with /var/log/ceph/cluster2-client.mirror.72700.log.gz 2024-09-12T06:59:45.789 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.40185.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84837.log 2024-09-12T06:59:45.793 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.45844.log: -- /var/log/ceph/cluster2-client.mirror.48572.log 2024-09-12T06:59:45.793 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80705.log 2024-09-12T06:59:45.793 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.48572.log: /var/log/ceph/cluster1-client.mirror.84837.log: 86.6% 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48572.log.gz 2024-09-12T06:59:45.793 INFO:teuthology.orchestra.run.smithi152.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.84837.log.gz 2024-09-12T06:59:45.793 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41242.log 2024-09-12T06:59:45.793 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83111.log 2024-09-12T06:59:45.793 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.41242.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80754.log 2024-09-12T06:59:45.795 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.83111.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.83111.log.gz 2024-09-12T06:59:45.795 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40605.log 2024-09-12T06:59:45.795 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.80705.log: /var/log/ceph/cluster2-client.mirror.80754.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.80705.log.gz 2024-09-12T06:59:45.795 INFO:teuthology.orchestra.run.smithi152.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.80754.log.gz 2024-09-12T06:59:45.796 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85872.log 2024-09-12T06:59:45.805 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.40605.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59360.log 2024-09-12T06:59:45.805 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.85872.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39827.log 2024-09-12T06:59:45.806 INFO:teuthology.orchestra.run.smithi152.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.85872.log.gz 2024-09-12T06:59:45.806 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.59360.log: gzip 1.3% -- replaced with /var/log/ceph/cluster2-client.mirror.59360.log.gz 2024-09-12T06:59:45.806 INFO:teuthology.orchestra.run.smithi152.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41479.log 2024-09-12T06:59:45.806 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82471.log 2024-09-12T06:59:45.806 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.41479.log: /var/log/ceph/cluster2-client.mirror.39827.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84747.log 2024-09-12T06:59:45.806 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.82471.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85175.log 2024-09-12T06:59:45.806 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.84747.log: 88.3% 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.84747.log.gz 2024-09-12T06:59:45.806 INFO:teuthology.orchestra.run.smithi152.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.82471.log.gz 2024-09-12T06:59:45.806 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76218.log 2024-09-12T06:59:45.806 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.85175.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.32757.log 2024-09-12T06:59:45.806 INFO:teuthology.orchestra.run.smithi152.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.85175.log.gz 2024-09-12T06:59:45.806 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.76218.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41504.log 2024-09-12T06:59:45.806 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.admin.32757.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.32757.log.gz 2024-09-12T06:59:45.808 INFO:teuthology.orchestra.run.smithi152.stderr: 91.7% -- replaced with /var/log/ceph/cluster1-client.mirror.76218.log.gz 2024-09-12T06:59:45.808 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81070.log 2024-09-12T06:59:45.808 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.41504.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87464.log 2024-09-12T06:59:45.809 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.81070.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82704.log 2024-09-12T06:59:45.810 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.87464.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.81070.log.gz 2024-09-12T06:59:45.811 INFO:teuthology.orchestra.run.smithi152.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.87464.log.gz 2024-09-12T06:59:45.811 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88507.log 2024-09-12T06:59:45.811 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.82704.log: -- /var/log/ceph/cluster2-client.mirror.38447.log 2024-09-12T06:59:45.811 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.88507.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.88507.log.gz 2024-09-12T06:59:45.813 INFO:teuthology.orchestra.run.smithi152.stderr: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.82704.log.gz 2024-09-12T06:59:45.813 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45388.log 2024-09-12T06:59:45.814 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.38447.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88709.log 2024-09-12T06:59:45.817 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42623.log 2024-09-12T06:59:45.817 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.88709.log: /var/log/ceph/cluster2-client.mirror.45388.log: 83.4% -- replaced with /var/log/ceph/cluster2-client.mirror.88709.log.gz 2024-09-12T06:59:45.817 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79702.log 2024-09-12T06:59:45.817 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83712.log 2024-09-12T06:59:45.817 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.79702.log: /var/log/ceph/cluster2-client.mirror.42623.log: gzip -5 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.79702.log.gz 2024-09-12T06:59:45.817 INFO:teuthology.orchestra.run.smithi152.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.38875.log 2024-09-12T06:59:45.820 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.83712.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.83712.log.gz 2024-09-12T06:59:45.820 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84882.log 2024-09-12T06:59:45.820 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38560.log 2024-09-12T06:59:45.820 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.84882.log: /var/log/ceph/cluster2-client.mirror.38875.log: 86.9%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39859.log 2024-09-12T06:59:45.822 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.38560.log: -- replaced with /var/log/ceph/cluster1-client.mirror.84882.log.gz 2024-09-12T06:59:45.822 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51848.log 2024-09-12T06:59:45.822 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.39859.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80227.log 2024-09-12T06:59:45.824 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.51848.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87748.log 2024-09-12T06:59:45.825 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.80227.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.80227.log.gz 2024-09-12T06:59:45.829 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78048.log 2024-09-12T06:59:45.832 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.87748.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.87748.log.gz 2024-09-12T06:59:45.832 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70915.log 2024-09-12T06:59:45.832 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.78048.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.78048.log.gz 2024-09-12T06:59:45.833 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68848.log 2024-09-12T06:59:45.833 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.70915.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.70915.log.gz 2024-09-12T06:59:45.835 INFO:teuthology.orchestra.run.smithi152.stderr: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.75089.log.gz 2024-09-12T06:59:45.835 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83834.log 2024-09-12T06:59:45.835 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.68848.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67228.log 2024-09-12T06:59:45.840 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.83834.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.83834.log.gz 2024-09-12T06:59:45.840 INFO:teuthology.orchestra.run.smithi152.stderr: 91.9% -- replaced with /var/log/ceph/cluster2-client.mirror.68848.log.gz 2024-09-12T06:59:45.841 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44094.log 2024-09-12T06:59:45.841 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72473.log 2024-09-12T06:59:45.841 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.44094.log: /var/log/ceph/cluster2-client.mirror.67228.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51322.log 2024-09-12T06:59:45.841 INFO:teuthology.orchestra.run.smithi152.stderr: 87.9%/var/log/ceph/cluster2-client.mirror.72473.log: -- replaced with /var/log/ceph/cluster2-client.mirror.67228.log.gz 2024-09-12T06:59:45.841 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.72473.log.gz 2024-09-12T06:59:45.841 INFO:teuthology.orchestra.run.smithi152.stderr: -- /var/log/ceph/cluster2-client.mirror.37788.log 2024-09-12T06:59:45.841 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56230.log 2024-09-12T06:59:45.841 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.37788.log: /var/log/ceph/cluster2-client.mirror.51322.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50011.log 2024-09-12T06:59:45.848 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.56230.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.38560.log.gz 2024-09-12T06:59:45.848 INFO:teuthology.orchestra.run.smithi152.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47658.log.gz 2024-09-12T06:59:45.849 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36296.log 2024-09-12T06:59:45.849 INFO:teuthology.orchestra.run.smithi152.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.41242.log.gz 2024-09-12T06:59:45.850 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.50011.log: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.41479.log.gz 2024-09-12T06:59:45.850 INFO:teuthology.orchestra.run.smithi152.stderr: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.45844.log.gz 2024-09-12T06:59:45.851 INFO:teuthology.orchestra.run.smithi152.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.42623.log.gz 2024-09-12T06:59:45.853 INFO:teuthology.orchestra.run.smithi152.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.40185.log.gz 2024-09-12T06:59:45.985 INFO:teuthology.orchestra.run.smithi152.stderr: 69.0% -- replaced with /var/log/ceph/cluster2-client.mirror.37788.log.gz 2024-09-12T06:59:45.985 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67718.log 2024-09-12T06:59:45.985 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.admin.36296.log: 0.0% 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.43531.log.gz 2024-09-12T06:59:45.985 INFO:teuthology.orchestra.run.smithi152.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.36296.log.gz 2024-09-12T06:59:45.985 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82166.log 2024-09-12T06:59:45.986 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.67718.log: 91.7% -- replaced with /var/log/ceph/cluster1-client.mirror.67718.log.gz 2024-09-12T06:59:45.986 INFO:teuthology.orchestra.run.smithi152.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.38875.log.gz 2024-09-12T06:59:45.986 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36422.log 2024-09-12T06:59:45.986 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.82166.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.82166.log.gz 2024-09-12T06:59:45.986 INFO:teuthology.orchestra.run.smithi152.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.39827.log.gz 2024-09-12T06:59:45.986 INFO:teuthology.orchestra.run.smithi152.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.39859.log.gz 2024-09-12T06:59:45.986 INFO:teuthology.orchestra.run.smithi152.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45388.log.gz 2024-09-12T06:59:45.986 INFO:teuthology.orchestra.run.smithi152.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.45409.log.gz 2024-09-12T06:59:45.986 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41057.log 2024-09-12T06:59:45.986 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.admin.36422.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36422.log.gz 2024-09-12T06:59:45.986 INFO:teuthology.orchestra.run.smithi152.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.68504.log.gz 2024-09-12T06:59:45.986 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85242.log 2024-09-12T06:59:45.987 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.41057.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74689.log 2024-09-12T06:59:45.987 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.85242.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.85242.log.gz 2024-09-12T06:59:45.987 INFO:teuthology.orchestra.run.smithi152.stderr: 95.3% -- replaced with /var/log/ceph/cluster2-client.mirror.50178.log.gz 2024-09-12T06:59:45.987 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41011.log 2024-09-12T06:59:45.987 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.74689.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82024.log 2024-09-12T06:59:45.987 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.41011.log: 91.7% -- replaced with /var/log/ceph/cluster1-client.mirror.74689.log.gz 2024-09-12T06:59:45.987 INFO:teuthology.orchestra.run.smithi152.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.40242.log.gz 2024-09-12T06:59:45.987 INFO:teuthology.orchestra.run.smithi152.stderr: 92.6% -- replaced with /var/log/ceph/cluster1-client.mirror.40309.log.gz 2024-09-12T06:59:45.987 INFO:teuthology.orchestra.run.smithi152.stderr: 92.6% -- replaced with /var/log/ceph/cluster1-client.mirror.40625.log.gz 2024-09-12T06:59:45.987 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72952.log 2024-09-12T06:59:45.987 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.82024.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44790.log 2024-09-12T06:59:45.987 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.72952.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.72952.log.gz 2024-09-12T06:59:45.987 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86120.log 2024-09-12T06:59:45.988 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.44790.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60468.log 2024-09-12T06:59:45.988 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.86120.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.86120.log.gz 2024-09-12T06:59:45.988 INFO:teuthology.orchestra.run.smithi152.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.82024.log.gz 2024-09-12T06:59:45.988 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87127.log 2024-09-12T06:59:45.988 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.60468.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41593.log 2024-09-12T06:59:45.988 INFO:teuthology.orchestra.run.smithi152.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.60468.log.gz 2024-09-12T06:59:45.988 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.87127.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55348.log 2024-09-12T06:59:45.988 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.41593.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.87127.log.gz 2024-09-12T06:59:45.988 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39049.log 2024-09-12T06:59:45.988 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36208.log 2024-09-12T06:59:45.988 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.39049.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65309.log 2024-09-12T06:59:45.988 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.admin.36208.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36208.log.gz 2024-09-12T06:59:45.988 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.55348.log: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55348.log.gz 2024-09-12T06:59:45.989 INFO:teuthology.orchestra.run.smithi152.stderr: 94.0% -- replaced with /var/log/ceph/cluster2.audit.log.gz 2024-09-12T06:59:45.989 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46605.log 2024-09-12T06:59:45.989 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.65309.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35960.log 2024-09-12T06:59:45.989 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.46605.log: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.65309.log.gz 2024-09-12T06:59:45.989 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37094.log 2024-09-12T06:59:45.989 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64187.log 2024-09-12T06:59:45.989 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.admin.35960.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35960.log.gz 2024-09-12T06:59:45.989 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.admin.37094.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37094.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85040.log 2024-09-12T06:59:45.989 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-09-12T06:59:45.989 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.64187.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45254.log 2024-09-12T06:59:45.989 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.85040.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.85040.log.gz 2024-09-12T06:59:45.989 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41031.log 2024-09-12T06:59:45.990 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.45254.log: 91.3% -- replaced with /var/log/ceph/cluster1-client.mirror.64187.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45606.log 2024-09-12T06:59:45.990 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-09-12T06:59:45.990 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5/var/log/ceph/cluster1-client.mirror.41031.log: --verbose -- /var/log/ceph/cluster1-client.mirror.84410.log 2024-09-12T06:59:45.990 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88323.log 2024-09-12T06:59:45.990 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.84410.log: /var/log/ceph/cluster2-client.mirror.45606.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.84410.log.gz 2024-09-12T06:59:45.990 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40943.log 2024-09-12T06:59:45.990 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.88323.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88385.log 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.88323.log.gz 2024-09-12T06:59:45.990 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-09-12T06:59:45.990 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.40943.log: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.60189.log.gz 2024-09-12T06:59:45.990 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44072.log 2024-09-12T06:59:45.990 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.88385.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79223.log 2024-09-12T06:59:45.990 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.44072.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41335.log 2024-09-12T06:59:45.991 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose/var/log/ceph/cluster1-client.mirror.79223.log: -- /var/log/ceph/cluster1-client.mirror.40217.log 2024-09-12T06:59:45.991 INFO:teuthology.orchestra.run.smithi152.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.79223.log.gz 2024-09-12T06:59:45.991 INFO:teuthology.orchestra.run.smithi152.stderr: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.88385.log.gz 2024-09-12T06:59:45.991 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.40217.log: gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.41335.log: -- /var/log/ceph/cluster2-client.mirror.52240.log 2024-09-12T06:59:45.991 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64931.log 2024-09-12T06:59:45.991 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.52240.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39792.log 2024-09-12T06:59:45.991 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.64931.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89484.log 2024-09-12T06:59:45.991 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.39792.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84545.log 2024-09-12T06:59:45.991 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.89484.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.89484.log.gz 2024-09-12T06:59:45.991 INFO:teuthology.orchestra.run.smithi152.stderr: 91.7% -- replaced with /var/log/ceph/cluster2-client.mirror.64931.log.gz 2024-09-12T06:59:45.991 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62681.log 2024-09-12T06:59:45.991 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.84545.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.84545.log.gz 2024-09-12T06:59:45.991 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50825.log 2024-09-12T06:59:45.992 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.62681.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36842.log 2024-09-12T06:59:45.992 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.50825.log: 95.9% -- replaced with /var/log/ceph/cluster2-client.mirror.62681.log.gz 2024-09-12T06:59:45.992 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46807.log 2024-09-12T06:59:45.992 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.admin.36842.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36842.log.gz 2024-09-12T06:59:45.992 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38525.log 2024-09-12T06:59:45.992 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.46807.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63403.log 2024-09-12T06:59:45.992 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.38525.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59402.log 2024-09-12T06:59:45.992 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.63403.log: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63403.log.gz 2024-09-12T06:59:45.992 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86075.log 2024-09-12T06:59:45.992 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.59402.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.59402.log.gz 2024-09-12T06:59:45.992 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84455.log 2024-09-12T06:59:45.992 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.86075.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.86075.log.gz 2024-09-12T06:59:45.993 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84025.log 2024-09-12T06:59:45.993 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.84455.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.84455.log.gz 2024-09-12T06:59:45.997 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76596.log 2024-09-12T06:59:45.997 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.84025.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.84025.log.gz 2024-09-12T06:59:45.997 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81770.log 2024-09-12T06:59:45.997 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.76596.log: 91.4% -- replaced with /var/log/ceph/cluster1-client.mirror.76596.log.gz 2024-09-12T06:59:45.997 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48712.log 2024-09-12T06:59:45.997 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.81770.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.81770.log.gz 2024-09-12T06:59:45.998 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73947.log 2024-09-12T06:59:45.998 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.48712.log: 84.3% -- replaced with /var/log/ceph/cluster2-client.mirror.48712.log.gz 2024-09-12T06:59:45.998 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39884.log 2024-09-12T06:59:46.005 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.73947.log: gzip -5 --verbose -- /var/log/ceph/cluster2-mon.a.log 2024-09-12T06:59:46.012 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.39884.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63423.log 2024-09-12T06:59:46.019 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-mon.a.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44923.log 2024-09-12T06:59:46.032 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80634.log 2024-09-12T06:59:46.034 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.63423.log: /var/log/ceph/cluster1-client.mirror.44923.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.63423.log.gz 2024-09-12T06:59:46.035 INFO:teuthology.orchestra.run.smithi152.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.73947.log.gz 2024-09-12T06:59:46.050 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48212.log 2024-09-12T06:59:46.051 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.80634.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.80634.log.gz 2024-09-12T06:59:46.056 INFO:teuthology.orchestra.run.smithi152.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.53334.log.gz 2024-09-12T06:59:46.061 INFO:teuthology.orchestra.run.smithi152.stderr: 95.9% -- replaced with /var/log/ceph/cluster2-client.mirror.51322.log.gz 2024-09-12T06:59:46.061 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85152.log 2024-09-12T06:59:46.061 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.48212.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79822.log 2024-09-12T06:59:46.061 INFO:teuthology.orchestra.run.smithi152.stderr: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48212.log.gz 2024-09-12T06:59:46.061 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.85152.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69933.log 2024-09-12T06:59:46.063 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.79822.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.79822.log.gz 2024-09-12T06:59:46.063 INFO:teuthology.orchestra.run.smithi152.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.mirror.85152.log.gz 2024-09-12T06:59:46.063 INFO:teuthology.orchestra.run.smithi152.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.40605.log.gz 2024-09-12T06:59:46.063 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87620.log 2024-09-12T06:59:46.065 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.69933.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.69933.log.gz 2024-09-12T06:59:46.065 INFO:teuthology.orchestra.run.smithi152.stderr: 92.9% -- replaced with /var/log/ceph/cluster2-client.mirror.40652.log.gz 2024-09-12T06:59:46.065 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49580.log 2024-09-12T06:59:46.066 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.87620.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.87620.log.gz 2024-09-12T06:59:46.066 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55371.log 2024-09-12T06:59:46.070 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.49580.log: 88.1% 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.41335.log.gz 2024-09-12T06:59:46.070 INFO:teuthology.orchestra.run.smithi152.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.38525.log.gz 2024-09-12T06:59:46.070 INFO:teuthology.orchestra.run.smithi152.stderr: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.40943.log.gz 2024-09-12T06:59:46.079 INFO:teuthology.orchestra.run.smithi152.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51848.log.gz 2024-09-12T06:59:46.080 INFO:teuthology.orchestra.run.smithi152.stderr: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.40217.log.gz 2024-09-12T06:59:46.083 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51825.log 2024-09-12T06:59:46.083 INFO:teuthology.orchestra.run.smithi152.stderr: 84.3% -- replaced with /var/log/ceph/cluster2-client.mirror.46807.log.gz 2024-09-12T06:59:46.083 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.55371.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45254.log.gz 2024-09-12T06:59:46.083 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45056.log 2024-09-12T06:59:46.083 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.51825.log: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51825.log.gz 2024-09-12T06:59:46.084 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88920.log 2024-09-12T06:59:46.086 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.45056.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60090.log 2024-09-12T06:59:46.086 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.88920.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.88920.log.gz 2024-09-12T06:59:46.087 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5/var/log/ceph/cluster2-client.mirror.60090.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.60090.log.gz 2024-09-12T06:59:46.090 INFO:teuthology.orchestra.run.smithi152.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.82308.log 2024-09-12T06:59:46.090 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72601.log 2024-09-12T06:59:46.090 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.82308.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.82308.log.gz 2024-09-12T06:59:46.101 INFO:teuthology.orchestra.run.smithi152.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50011.log.gz 2024-09-12T06:59:46.111 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose 93.1% -- replaced with /var/log/ceph/cluster2-client.mirror.44989.log.gz 2024-09-12T06:59:46.111 INFO:teuthology.orchestra.run.smithi152.stderr: -- /var/log/ceph/cluster2-client.mirror.88365.log 2024-09-12T06:59:46.111 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.72601.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85355.log 2024-09-12T06:59:46.111 INFO:teuthology.orchestra.run.smithi152.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.72601.log.gz 2024-09-12T06:59:46.111 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.88365.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43980.log 2024-09-12T06:59:46.111 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.85355.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.88365.log.gz 2024-09-12T06:59:46.112 INFO:teuthology.orchestra.run.smithi152.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.85355.log.gz 2024-09-12T06:59:46.112 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81403.log 2024-09-12T06:59:46.112 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose --/var/log/ceph/cluster1-client.mirror.43980.log: /var/log/ceph/cluster1-client.mirror.75022.log 2024-09-12T06:59:46.112 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86566.log 2024-09-12T06:59:46.112 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.81403.log: /var/log/ceph/cluster1-client.mirror.75022.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.75022.log.gz 2024-09-12T06:59:46.112 INFO:teuthology.orchestra.run.smithi152.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.81403.log.gz 2024-09-12T06:59:46.112 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38388.log 2024-09-12T06:59:46.112 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.86566.log: 61.3%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61783.log 2024-09-12T06:59:46.112 INFO:teuthology.orchestra.run.smithi152.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.86566.log.gz 2024-09-12T06:59:46.112 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.admin.38388.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45693.log 2024-09-12T06:59:46.112 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.61783.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41277.log 2024-09-12T06:59:46.112 INFO:teuthology.orchestra.run.smithi152.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.61783.log.gz 2024-09-12T06:59:46.112 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.45693.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87505.log 2024-09-12T06:59:46.124 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.41277.log: 94.2% -- replaced with /var/log/ceph/cluster1-client.mirror.38002.log.gz 2024-09-12T06:59:46.125 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81354.log 2024-09-12T06:59:46.125 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.87505.log: gzip -5 --verbose 83.4% -- replaced with /var/log/ceph/cluster2-client.mirror.87505.log.gz 2024-09-12T06:59:46.125 INFO:teuthology.orchestra.run.smithi152.stderr: -- /var/log/ceph/cluster2-client.admin.32837.log 2024-09-12T06:59:46.125 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.81354.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89507.log 2024-09-12T06:59:46.125 INFO:teuthology.orchestra.run.smithi152.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.81354.log.gz/var/log/ceph/cluster2-client.admin.32837.log: 2024-09-12T06:59:46.125 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.32837.log.gz 2024-09-12T06:59:46.125 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63781.log 2024-09-12T06:59:46.125 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.89507.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45649.log 2024-09-12T06:59:46.125 INFO:teuthology.orchestra.run.smithi152.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.89507.log.gz 2024-09-12T06:59:46.125 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.63781.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70511.log 2024-09-12T06:59:46.125 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.45649.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81332.log 2024-09-12T06:59:46.126 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.70511.log: -- /var/log/ceph/cluster2-client.mirror.56918.log 2024-09-12T06:59:46.126 INFO:teuthology.orchestra.run.smithi152.stderr: 91.9% -- replaced with /var/log/ceph/cluster2-client.mirror.63781.log.gz 2024-09-12T06:59:46.126 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.81332.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61875.log 2024-09-12T06:59:46.126 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.56918.log: 1.3% -- replaced with /var/log/ceph/cluster2-client.mirror.56918.log.gz 2024-09-12T06:59:46.126 INFO:teuthology.orchestra.run.smithi152.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.81332.log.gz 2024-09-12T06:59:46.126 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55583.log 2024-09-12T06:59:46.126 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.61875.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85017.log 2024-09-12T06:59:46.126 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.55583.log: 84.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55583.log.gz 2024-09-12T06:59:46.126 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85985.log 2024-09-12T06:59:46.126 INFO:teuthology.orchestra.run.smithi152.stderr: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.61875.log.gz 2024-09-12T06:59:46.126 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.85017.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.85017.log.gz 2024-09-12T06:59:46.126 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83354.log 2024-09-12T06:59:46.126 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.85985.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83609.log 2024-09-12T06:59:46.127 INFO:teuthology.orchestra.run.smithi152.stderr: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.85985.log.gz 2024-09-12T06:59:46.127 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.83354.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71704.log 2024-09-12T06:59:46.127 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.83609.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.83354.log.gz 2024-09-12T06:59:46.127 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69268.log 2024-09-12T06:59:46.127 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.71704.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.83609.log.gz 2024-09-12T06:59:46.129 INFO:teuthology.orchestra.run.smithi152.stderr: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.71704.log.gz 2024-09-12T06:59:46.129 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-osd.1.log 2024-09-12T06:59:46.129 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63059.log 2024-09-12T06:59:46.130 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.69268.log: /var/log/ceph/cluster2-osd.1.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54088.log 2024-09-12T06:59:46.141 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.63059.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.45606.log.gz 2024-09-12T06:59:46.142 INFO:teuthology.orchestra.run.smithi152.stderr: 88.7% -- replaced with /var/log/ceph/cluster1-client.mirror.43980.log.gz 2024-09-12T06:59:46.147 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86345.log 2024-09-12T06:59:46.148 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.54088.log: 84.4% -- replaced with /var/log/ceph/cluster2-client.mirror.54088.log.gz 2024-09-12T06:59:46.163 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37423.log 2024-09-12T06:59:46.163 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.86345.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.86345.log.gz 2024-09-12T06:59:46.170 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81141.log 2024-09-12T06:59:46.170 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.admin.37423.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37423.log.gz 2024-09-12T06:59:46.197 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39517.log 2024-09-12T06:59:46.198 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.81141.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.81141.log.gz 2024-09-12T06:59:46.199 INFO:teuthology.orchestra.run.smithi152.stderr: 93.1% -- replaced with /var/log/ceph/cluster2-client.mirror.44903.log.gz 2024-09-12T06:59:46.200 INFO:teuthology.orchestra.run.smithi152.stderr: 96.2% -- replaced with /var/log/ceph/cluster2-client.mirror.50825.log.gz 2024-09-12T06:59:46.201 INFO:teuthology.orchestra.run.smithi152.stderr: 93.0% -- replaced with /var/log/ceph/cluster2-client.mirror.44856.log.gz 2024-09-12T06:59:46.202 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81577.log 2024-09-12T06:59:46.205 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.39517.log: /var/log/ceph/cluster2-client.mirror.81577.log: gzip -5 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.81577.log.gz 2024-09-12T06:59:46.205 INFO:teuthology.orchestra.run.smithi152.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.41277.log.gz 2024-09-12T06:59:46.207 INFO:teuthology.orchestra.run.smithi152.stderr: --verbose -- /var/log/ceph/cluster1-client.mirror.67340.log 2024-09-12T06:59:46.207 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53311.log 2024-09-12T06:59:46.210 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.67340.log: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67340.log.gz 2024-09-12T06:59:46.210 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88603.log 2024-09-12T06:59:46.213 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.53311.log: 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.53311.log.gz 2024-09-12T06:59:46.213 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36335.log 2024-09-12T06:59:46.213 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.88603.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.88603.log.gz 2024-09-12T06:59:46.213 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85287.log 2024-09-12T06:59:46.213 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.admin.36335.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36335.log.gz 2024-09-12T06:59:46.214 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.70511.log.gz 2024-09-12T06:59:46.218 INFO:teuthology.orchestra.run.smithi152.stderr: -- /var/log/ceph/cluster2-client.mirror.88728.log 2024-09-12T06:59:46.219 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose/var/log/ceph/cluster1-client.mirror.85287.log: -- /var/log/ceph/cluster1-client.mirror.84702.log 2024-09-12T06:59:46.219 INFO:teuthology.orchestra.run.smithi152.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.85287.log.gz 2024-09-12T06:59:46.219 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.88728.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44025.log 2024-09-12T06:59:46.219 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.84702.log: 87.0% 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.88728.log.gz 2024-09-12T06:59:46.219 INFO:teuthology.orchestra.run.smithi152.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.84702.log.gz 2024-09-12T06:59:46.219 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59425.log 2024-09-12T06:59:46.219 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82095.log 2024-09-12T06:59:46.223 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.59425.log: /var/log/ceph/cluster1-client.mirror.44025.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66612.log 2024-09-12T06:59:46.223 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.82095.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.38388.log.gz 2024-09-12T06:59:46.223 INFO:teuthology.orchestra.run.smithi152.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.82095.log.gz 2024-09-12T06:59:46.223 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37771.log 2024-09-12T06:59:46.225 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.66612.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66612.log.gz 2024-09-12T06:59:46.225 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38713.log 2024-09-12T06:59:46.225 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.37771.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83783.log 2024-09-12T06:59:46.226 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.admin.38713.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39927.log 2024-09-12T06:59:46.229 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.83783.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.83783.log.gz 2024-09-12T06:59:46.229 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50155.log 2024-09-12T06:59:46.231 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.39927.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45581.log 2024-09-12T06:59:46.231 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.50155.log: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50155.log.gz 2024-09-12T06:59:46.233 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36968.log 2024-09-12T06:59:46.233 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.45581.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.32624.log 2024-09-12T06:59:46.233 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.admin.36968.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36968.log.gz 2024-09-12T06:59:46.235 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88165.log 2024-09-12T06:59:46.235 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.admin.32624.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.32624.log.gz 2024-09-12T06:59:46.235 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69612.log 2024-09-12T06:59:46.245 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.88165.log: 96.6%gzip -- replaced with /var/log/ceph/cluster2-client.mirror.40694.log.gz 2024-09-12T06:59:46.245 INFO:teuthology.orchestra.run.smithi152.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66434.log 2024-09-12T06:59:46.246 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.69612.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.44072.log.gz 2024-09-12T06:59:46.246 INFO:teuthology.orchestra.run.smithi152.stderr: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.44391.log.gz 2024-09-12T06:59:46.250 INFO:teuthology.orchestra.run.smithi152.stderr: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.69612.log.gz 2024-09-12T06:59:46.261 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85917.log 2024-09-12T06:59:46.261 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.66434.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.66434.log.gz 2024-09-12T06:59:46.270 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43647.log 2024-09-12T06:59:46.270 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.85917.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.85917.log.gz 2024-09-12T06:59:46.277 INFO:teuthology.orchestra.run.smithi152.stderr: 93.1% -- replaced with /var/log/ceph/cluster2-client.mirror.41504.log.gz 2024-09-12T06:59:46.280 INFO:teuthology.orchestra.run.smithi152.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.41057.log.gz 2024-09-12T06:59:46.283 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82918.log 2024-09-12T06:59:46.283 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.admin.43647.log: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.39049.log.gz 2024-09-12T06:59:46.285 INFO:teuthology.orchestra.run.smithi152.stderr: 93.1% -- replaced with /var/log/ceph/cluster2-client.mirror.41551.log.gz 2024-09-12T06:59:46.292 INFO:teuthology.orchestra.run.smithi152.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.56230.log.gz 2024-09-12T06:59:46.293 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68482.log 2024-09-12T06:59:46.295 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.82918.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.82918.log.gz 2024-09-12T06:59:46.295 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47296.log 2024-09-12T06:59:46.297 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.68482.log: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68482.log.gz 2024-09-12T06:59:46.298 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77927.log 2024-09-12T06:59:46.308 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.47296.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39540.log 2024-09-12T06:59:46.310 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.77927.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.77927.log.gz 2024-09-12T06:59:46.312 INFO:teuthology.orchestra.run.smithi152.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.44590.log.gz 2024-09-12T06:59:46.321 INFO:teuthology.orchestra.run.smithi152.stderr: 88.8% -- replaced with /var/log/ceph/cluster1-client.admin.43647.log.gz 2024-09-12T06:59:46.321 INFO:teuthology.orchestra.run.smithi152.stderr: 88.5% -- replaced with /var/log/ceph/cluster1-client.admin.38713.log.gz 2024-09-12T06:59:46.321 INFO:teuthology.orchestra.run.smithi152.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.45581.log.gz 2024-09-12T06:59:46.322 INFO:teuthology.orchestra.run.smithi152.stderr: 70.7% -- replaced with /var/log/ceph/cluster2-client.mirror.37771.log.gz 2024-09-12T06:59:46.323 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45451.log 2024-09-12T06:59:46.325 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.39540.log: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.55371.log.gz 2024-09-12T06:59:46.326 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42472.log 2024-09-12T06:59:46.329 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.45451.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86502.log 2024-09-12T06:59:46.329 INFO:teuthology.orchestra.run.smithi152.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49580.log.gz 2024-09-12T06:59:46.329 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39692.log 2024-09-12T06:59:46.330 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.86502.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.86502.log.gz 2024-09-12T06:59:46.332 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.42472.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82662.log 2024-09-12T06:59:46.332 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.admin.39692.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60567.log 2024-09-12T06:59:46.332 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.82662.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39092.log 2024-09-12T06:59:46.332 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.60567.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.82662.log.gz 2024-09-12T06:59:46.335 INFO:teuthology.orchestra.run.smithi152.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.59425.log.gz 2024-09-12T06:59:46.338 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66459.log 2024-09-12T06:59:46.338 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.39092.log: 96.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44094.log.gz 2024-09-12T06:59:46.338 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.90331.log 2024-09-12T06:59:46.339 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.66459.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88750.log 2024-09-12T06:59:46.339 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.admin.90331.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.90331.log.gz 2024-09-12T06:59:46.339 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85895.log 2024-09-12T06:59:46.339 INFO:teuthology.orchestra.run.smithi152.stderr: 91.8%/var/log/ceph/cluster2-client.mirror.88750.log: -- replaced with /var/log/ceph/cluster2-client.mirror.66459.log.gz 2024-09-12T06:59:46.341 INFO:teuthology.orchestra.run.smithi152.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.88750.log.gz 2024-09-12T06:59:46.342 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88982.log 2024-09-12T06:59:46.342 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86187.log 2024-09-12T06:59:46.342 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.88982.log: /var/log/ceph/cluster1-client.mirror.85895.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.85895.log.gz 2024-09-12T06:59:46.342 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89193.log 2024-09-12T06:59:46.342 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.86187.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.86187.log.gz 2024-09-12T06:59:46.342 INFO:teuthology.orchestra.run.smithi152.stderr: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.88982.log.gz 2024-09-12T06:59:46.342 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84770.log 2024-09-12T06:59:46.344 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.89193.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.89193.log.gz 2024-09-12T06:59:46.344 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78778.log 2024-09-12T06:59:46.344 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.84770.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.84770.log.gz 2024-09-12T06:59:46.346 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.78778.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.78778.log.gz 2024-09-12T06:59:46.346 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73880.log 2024-09-12T06:59:46.346 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73539.log 2024-09-12T06:59:46.346 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.73880.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.73880.log.gz 2024-09-12T06:59:46.351 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.73539.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44700.log 2024-09-12T06:59:46.351 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74577.log 2024-09-12T06:59:46.351 INFO:teuthology.orchestra.run.smithi152.stderr: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.73539.log.gz 2024-09-12T06:59:46.351 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.44700.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88473.log 2024-09-12T06:59:46.351 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.74577.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44457.log 2024-09-12T06:59:46.351 INFO:teuthology.orchestra.run.smithi152.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.74577.log.gz 2024-09-12T06:59:46.351 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.88473.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48235.log 2024-09-12T06:59:46.351 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.44457.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86277.log 2024-09-12T06:59:46.358 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.48235.log: 88.1% 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.45649.log.gz 2024-09-12T06:59:46.358 INFO:teuthology.orchestra.run.smithi152.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.45693.log.gz 2024-09-12T06:59:46.359 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85625.log 2024-09-12T06:59:46.366 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.86277.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74191.log 2024-09-12T06:59:46.366 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.85625.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.85625.log.gz 2024-09-12T06:59:46.372 INFO:teuthology.orchestra.run.smithi152.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.86277.log.gz 2024-09-12T06:59:46.387 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73183.log 2024-09-12T06:59:46.387 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.74191.log: 93.0% -- replaced with /var/log/ceph/cluster1-client.mirror.41031.log.gz 2024-09-12T06:59:46.387 INFO:teuthology.orchestra.run.smithi152.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.74191.log.gz 2024-09-12T06:59:46.389 INFO:teuthology.orchestra.run.smithi152.stderr: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.45056.log.gz 2024-09-12T06:59:46.403 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88036.log 2024-09-12T06:59:46.412 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.73183.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70489.log 2024-09-12T06:59:46.417 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.88036.log: 95.7% 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.88036.log.gz 2024-09-12T06:59:46.426 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80178.log 2024-09-12T06:59:46.428 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.70489.log: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.69268.log.gz 2024-09-12T06:59:46.428 INFO:teuthology.orchestra.run.smithi152.stderr: 91.8% -- replaced with /var/log/ceph/cluster1-client.mirror.70489.log.gz 2024-09-12T06:59:46.429 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67740.log 2024-09-12T06:59:46.430 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.80178.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.80178.log.gz 2024-09-12T06:59:46.430 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62228.log 2024-09-12T06:59:46.430 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.67740.log: -- replaced with /var/log/ceph/cluster2-client.mirror.52240.log.gz 2024-09-12T06:59:46.434 INFO:teuthology.orchestra.run.smithi152.stderr: 94.9% -- replaced with /var/log/ceph/cluster2-client.mirror.67740.log.gz 2024-09-12T06:59:46.434 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61397.log 2024-09-12T06:59:46.434 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.62228.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.62228.log.gz 2024-09-12T06:59:46.437 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67673.log 2024-09-12T06:59:46.437 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.61397.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.42472.log.gz 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.45451.log.gz 2024-09-12T06:59:46.437 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-09-12T06:59:46.438 INFO:teuthology.orchestra.run.smithi152.stderr: 92.8% -- replaced with /var/log/ceph/cluster1-client.mirror.44790.log.gz 2024-09-12T06:59:46.438 INFO:teuthology.orchestra.run.smithi152.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.61397.log.gz 2024-09-12T06:59:46.441 INFO:teuthology.orchestra.run.smithi152.stderr: 93.0% -- replaced with /var/log/ceph/cluster2-client.mirror.41011.log.gz 2024-09-12T06:59:46.444 INFO:teuthology.orchestra.run.smithi152.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.39884.log.gz 2024-09-12T06:59:46.444 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47898.log 2024-09-12T06:59:46.446 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.67673.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67673.log.gz 2024-09-12T06:59:46.446 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86734.log 2024-09-12T06:59:46.458 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.47898.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85962.log 2024-09-12T06:59:46.471 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.86734.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83741.log 2024-09-12T06:59:46.471 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.85962.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.85962.log.gz 2024-09-12T06:59:46.472 INFO:teuthology.orchestra.run.smithi152.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.86734.log.gz 2024-09-12T06:59:46.473 INFO:teuthology.orchestra.run.smithi152.stderr: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.44025.log.gz 2024-09-12T06:59:46.479 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41528.log 2024-09-12T06:59:46.484 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.83741.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.39517.log.gz 2024-09-12T06:59:46.484 INFO:teuthology.orchestra.run.smithi152.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.60567.log.gz 2024-09-12T06:59:46.484 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45099.log 2024-09-12T06:59:46.484 INFO:teuthology.orchestra.run.smithi152.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.83741.log.gz 2024-09-12T06:59:46.484 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.41528.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40988.log 2024-09-12T06:59:46.486 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.45099.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70755.log 2024-09-12T06:59:46.486 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.40988.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64142.log 2024-09-12T06:59:46.487 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.70755.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.70755.log.gz 2024-09-12T06:59:46.494 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36800.log 2024-09-12T06:59:46.497 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.64142.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.64142.log.gz 2024-09-12T06:59:46.498 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65670.log 2024-09-12T06:59:46.498 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.admin.36800.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36800.log.gz 2024-09-12T06:59:46.500 INFO:teuthology.orchestra.run.smithi152.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.73183.log.gz 2024-09-12T06:59:46.500 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38637.log 2024-09-12T06:59:46.501 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.65670.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.65670.log.gz 2024-09-12T06:59:46.501 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67698.log 2024-09-12T06:59:46.501 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88015.log 2024-09-12T06:59:46.502 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.38637.log: /var/log/ceph/cluster2-client.mirror.67698.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69587.log 2024-09-12T06:59:46.502 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.88015.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.88015.log.gz 2024-09-12T06:59:46.505 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82144.log 2024-09-12T06:59:46.505 INFO:teuthology.orchestra.run.smithi152.stderr: 91.9% -- replaced with /var/log/ceph/cluster2-client.mirror.67698.log.gz 2024-09-12T06:59:46.505 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.69587.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72658.log 2024-09-12T06:59:46.506 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.82144.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69587.log.gz 2024-09-12T06:59:46.506 INFO:teuthology.orchestra.run.smithi152.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.82144.log.gz 2024-09-12T06:59:46.506 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49299.log 2024-09-12T06:59:46.508 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.72658.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76106.log 2024-09-12T06:59:46.508 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.49299.log: 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49299.log.gz 2024-09-12T06:59:46.508 INFO:teuthology.orchestra.run.smithi152.stderr: 92.1% -- replaced with /var/log/ceph/cluster2-client.mirror.72658.log.gz 2024-09-12T06:59:46.508 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84590.log 2024-09-12T06:59:46.509 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.76106.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.76106.log.gz 2024-09-12T06:59:46.509 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84500.log 2024-09-12T06:59:46.510 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.84590.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.84590.log.gz 2024-09-12T06:59:46.511 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75754.log 2024-09-12T06:59:46.511 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.84500.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.84500.log.gz 2024-09-12T06:59:46.512 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39582.log 2024-09-12T06:59:46.513 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.75754.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.75754.log.gz 2024-09-12T06:59:46.513 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39904.log 2024-09-12T06:59:46.518 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.39582.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36464.log 2024-09-12T06:59:46.518 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41714.log 2024-09-12T06:59:46.518 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.39904.log: /var/log/ceph/cluster2-client.admin.36464.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36464.log.gz 2024-09-12T06:59:46.518 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84387.log 2024-09-12T06:59:46.518 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.41714.log: -- /var/log/ceph/cluster2-client.mirror.87251.log 2024-09-12T06:59:46.518 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39029.log 2024-09-12T06:59:46.518 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.87251.log: /var/log/ceph/cluster1-client.mirror.84387.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.84387.log.gz 2024-09-12T06:59:46.519 INFO:teuthology.orchestra.run.smithi152.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.87251.log.gz 2024-09-12T06:59:46.519 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37754.log 2024-09-12T06:59:46.519 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.37754.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85760.log 2024-09-12T06:59:46.519 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.39029.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82186.log 2024-09-12T06:59:46.520 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.85760.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.85760.log.gz 2024-09-12T06:59:46.520 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39196.log 2024-09-12T06:59:46.521 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.82186.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46495.log 2024-09-12T06:59:46.521 INFO:teuthology.orchestra.run.smithi152.stderr: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.82186.log.gz 2024-09-12T06:59:46.524 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.admin.39196.log: /var/log/ceph/cluster2-client.mirror.46495.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63823.log 2024-09-12T06:59:46.524 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45321.log 2024-09-12T06:59:46.524 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose --/var/log/ceph/cluster2-client.mirror.63823.log: /var/log/ceph/cluster1-client.admin.40745.log 2024-09-12T06:59:46.526 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.45321.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65329.log 2024-09-12T06:59:46.526 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.admin.40745.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37737.log 2024-09-12T06:59:46.533 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.65329.log: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.65329.log.gz 2024-09-12T06:59:46.534 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62161.log 2024-09-12T06:59:46.534 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.37737.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.37902.log 2024-09-12T06:59:46.535 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.62161.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.62161.log.gz 2024-09-12T06:59:46.538 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.90121.log 2024-09-12T06:59:46.540 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.1.37902.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43314.log 2024-09-12T06:59:46.540 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.admin.90121.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.90121.log.gz 2024-09-12T06:59:46.540 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68084.log 2024-09-12T06:59:46.544 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.43314.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59769.log 2024-09-12T06:59:46.550 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.68084.log: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.88473.log.gz 2024-09-12T06:59:46.554 INFO:teuthology.orchestra.run.smithi152.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.63059.log.gz 2024-09-12T06:59:46.554 INFO:teuthology.orchestra.run.smithi152.stderr: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.68084.log.gz 2024-09-12T06:59:46.554 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42844.log 2024-09-12T06:59:46.554 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.59769.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36716.log 2024-09-12T06:59:46.554 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.42844.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68126.log 2024-09-12T06:59:46.554 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.admin.36716.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36716.log.gz 2024-09-12T06:59:46.554 INFO:teuthology.orchestra.run.smithi152.stderr: 91.7% -- replaced with /var/log/ceph/cluster2-client.mirror.59769.log.gz 2024-09-12T06:59:46.554 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.68126.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45142.log 2024-09-12T06:59:46.554 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63378.log 2024-09-12T06:59:46.595 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.45142.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42809.log 2024-09-12T06:59:46.596 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.63378.log: gzip -5 --verbose -- 93.1% -- replaced with /var/log/ceph/cluster2-client.mirror.68126.log.gz 2024-09-12T06:59:46.596 INFO:teuthology.orchestra.run.smithi152.stderr: /var/log/ceph/cluster2-client.mirror.83231.log 2024-09-12T06:59:46.596 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.42809.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45560.log 2024-09-12T06:59:46.596 INFO:teuthology.orchestra.run.smithi152.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.63378.log.gz 2024-09-12T06:59:46.596 INFO:teuthology.orchestra.run.smithi152.stderr:gzip/var/log/ceph/cluster2-client.mirror.83231.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.38002.log 2024-09-12T06:59:46.596 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.45560.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43616.log 2024-09-12T06:59:46.596 INFO:teuthology.orchestra.run.smithi152.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.83231.log.gz 2024-09-12T06:59:46.596 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.0.38002.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67263.log 2024-09-12T06:59:46.596 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.admin.43616.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60854.log 2024-09-12T06:59:46.596 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.67263.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49557.log 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.67263.log.gz 2024-09-12T06:59:46.596 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-09-12T06:59:46.596 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.60854.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87169.log 2024-09-12T06:59:46.596 INFO:teuthology.orchestra.run.smithi152.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.60854.log.gz 2024-09-12T06:59:46.597 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.49557.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81161.log 2024-09-12T06:59:46.597 INFO:teuthology.orchestra.run.smithi152.stderr: 84.3% -- replaced with /var/log/ceph/cluster2-client.mirror.49557.log.gz 2024-09-12T06:59:46.597 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose --/var/log/ceph/cluster2-client.mirror.87169.log: /var/log/ceph/cluster2-client.admin.37010.log 2024-09-12T06:59:46.597 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.81161.log: gzip 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.87169.log.gz 2024-09-12T06:59:46.597 INFO:teuthology.orchestra.run.smithi152.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.admin.41686.log 2024-09-12T06:59:46.597 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.admin.37010.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75475.log 2024-09-12T06:59:46.597 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37010.log.gz 2024-09-12T06:59:46.597 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.admin.41686.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76484.log 2024-09-12T06:59:46.597 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.75475.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44833.log 2024-09-12T06:59:46.597 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.76484.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.76484.log.gz 2024-09-12T06:59:46.597 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84635.log 2024-09-12T06:59:46.597 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.44833.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.81161.log.gz 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.75475.log.gz 2024-09-12T06:59:46.597 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-09-12T06:59:46.598 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38068.log 2024-09-12T06:59:46.598 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.84635.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81719.log 2024-09-12T06:59:46.598 INFO:teuthology.orchestra.run.smithi152.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.84635.log.gz 2024-09-12T06:59:46.598 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.38068.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64520.log 2024-09-12T06:59:46.598 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.81719.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36506.log 2024-09-12T06:59:46.598 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.64520.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44302.log 2024-09-12T06:59:46.598 INFO:teuthology.orchestra.run.smithi152.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.64520.log.gz 2024-09-12T06:59:46.598 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.admin.36506.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80340.log 2024-09-12T06:59:46.598 INFO:teuthology.orchestra.run.smithi152.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36506.log.gz/var/log/ceph/cluster1-client.mirror.44302.log: 2024-09-12T06:59:46.598 INFO:teuthology.orchestra.run.smithi152.stderr: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.81719.log.gz 2024-09-12T06:59:46.598 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69246.log 2024-09-12T06:59:46.598 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.80340.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37600.log 2024-09-12T06:59:46.598 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44005.log 2024-09-12T06:59:46.599 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.37600.log: /var/log/ceph/cluster1-client.mirror.69246.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.80340.log.gz 2024-09-12T06:59:46.599 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85265.log 2024-09-12T06:59:46.599 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35918.log 2024-09-12T06:59:46.599 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.85265.log: /var/log/ceph/cluster2-client.mirror.44005.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.85265.log.gz 91.8% -- replaced with /var/log/ceph/cluster1-client.mirror.69246.log.gz 2024-09-12T06:59:46.599 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-09-12T06:59:46.599 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63037.log 2024-09-12T06:59:46.599 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87019.log 2024-09-12T06:59:46.599 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.admin.35918.log: /var/log/ceph/cluster1-client.mirror.63037.log: 0.0% 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.39196.log.gz 2024-09-12T06:59:46.599 INFO:teuthology.orchestra.run.smithi152.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.35918.log.gz 2024-09-12T06:59:46.599 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75376.log 2024-09-12T06:59:46.599 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.87019.log: 88.9% -- replaced with /var/log/ceph/cluster1-client.admin.40745.log.gz 2024-09-12T06:59:46.599 INFO:teuthology.orchestra.run.smithi152.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.87019.log.gz 2024-09-12T06:59:46.599 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66501.log 2024-09-12T06:59:46.600 INFO:teuthology.orchestra.run.smithi152.stderr: 89.4%/var/log/ceph/cluster2-client.mirror.75376.log: -- replaced with /var/log/ceph/cluster2-client.mirror.45321.log.gz 2024-09-12T06:59:46.600 INFO:teuthology.orchestra.run.smithi152.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.75376.log.gz 2024-09-12T06:59:46.600 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62582.log 2024-09-12T06:59:46.600 INFO:teuthology.orchestra.run.smithi152.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-client.mirror.63037.log.gz 2024-09-12T06:59:46.600 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.66501.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.89791.log 2024-09-12T06:59:46.600 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.62582.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42344.log 2024-09-12T06:59:46.600 INFO:teuthology.orchestra.run.smithi152.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.62582.log.gz 2024-09-12T06:59:46.600 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.admin.89791.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63756.log 2024-09-12T06:59:46.600 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.42344.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.89791.log.gz 2024-09-12T06:59:46.600 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42941.log 2024-09-12T06:59:46.600 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.63756.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.63756.log.gz 2024-09-12T06:59:46.600 INFO:teuthology.orchestra.run.smithi152.stderr: 92.6%gzip -- replaced with /var/log/ceph/cluster1-client.mirror.44743.log.gz 2024-09-12T06:59:46.601 INFO:teuthology.orchestra.run.smithi152.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79609.log 2024-09-12T06:59:46.601 INFO:teuthology.orchestra.run.smithi152.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.42344.log.gz 2024-09-12T06:59:46.601 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.42941.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79793.log 2024-09-12T06:59:46.601 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.79609.log: 95.9% -- replaced with /var/log/ceph/cluster2-client.mirror.66501.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70444.log 2024-09-12T06:59:46.601 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-09-12T06:59:46.601 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36209.log 2024-09-12T06:59:46.601 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.70444.log: /var/log/ceph/cluster2-client.mirror.79793.log: 88.1% 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.42941.log.gz 2024-09-12T06:59:46.601 INFO:teuthology.orchestra.run.smithi152.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.70444.log.gz 2024-09-12T06:59:46.601 INFO:teuthology.orchestra.run.smithi152.stderr: 89.0% 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.79609.log.gz 2024-09-12T06:59:46.601 INFO:teuthology.orchestra.run.smithi152.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.42809.log.gz 2024-09-12T06:59:46.601 INFO:teuthology.orchestra.run.smithi152.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.43314.log.gz 2024-09-12T06:59:46.601 INFO:teuthology.orchestra.run.smithi152.stderr: 54.2% -- replaced with /var/log/ceph/cluster1-client.mirror.37600.log.gz 2024-09-12T06:59:46.601 INFO:teuthology.orchestra.run.smithi152.stderr: 3.5% -- replaced with /var/log/ceph/cluster2-client.mirror.37754.log.gz 2024-09-12T06:59:46.601 INFO:teuthology.orchestra.run.smithi152.stderr: 3.5% -- replaced with /var/log/ceph/cluster2-client.mirror.37737.log.gz 2024-09-12T06:59:46.602 INFO:teuthology.orchestra.run.smithi152.stderr: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.79793.log.gz 2024-09-12T06:59:46.602 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82542.log 2024-09-12T06:59:46.602 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.admin.36209.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36209.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65638.log 2024-09-12T06:59:46.602 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-09-12T06:59:46.602 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39947.log 2024-09-12T06:59:46.602 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.65638.log: /var/log/ceph/cluster2-client.mirror.82542.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89161.log 2024-09-12T06:59:46.602 INFO:teuthology.orchestra.run.smithi152.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.65638.log.gz 2024-09-12T06:59:46.602 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.39947.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.82542.log.gz 2024-09-12T06:59:46.602 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87810.log 2024-09-12T06:59:46.602 INFO:teuthology.orchestra.run.smithi152.stderr:gzip/var/log/ceph/cluster2-client.mirror.89161.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48002.log 2024-09-12T06:59:46.602 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.87810.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86962.log 2024-09-12T06:59:46.602 INFO:teuthology.orchestra.run.smithi152.stderr: 83.3% -- replaced with /var/log/ceph/cluster2-client.mirror.87810.log.gz 2024-09-12T06:59:46.602 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.48002.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.90499.log 2024-09-12T06:59:46.602 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.86962.log: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48002.log.gz 2024-09-12T06:59:46.603 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59789.log 2024-09-12T06:59:46.603 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.admin.90499.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.90499.log.gz 2024-09-12T06:59:46.603 INFO:teuthology.orchestra.run.smithi152.stderr: 91.2%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50348.log 2024-09-12T06:59:46.603 INFO:teuthology.orchestra.run.smithi152.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.89161.log.gz 2024-09-12T06:59:46.603 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.59789.log: 91.5% -- replaced with /var/log/ceph/cluster1-client.mirror.59789.log.gz 2024-09-12T06:59:46.603 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83447.log 2024-09-12T06:59:46.603 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.50348.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79654.log 2024-09-12T06:59:46.603 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.83447.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.83447.log.gz 2024-09-12T06:59:46.610 INFO:teuthology.orchestra.run.smithi152.stderr: 95.9% -- replaced with /var/log/ceph/cluster2-client.mirror.47296.log.gz 2024-09-12T06:59:46.610 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89044.log 2024-09-12T06:59:46.610 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84860.log 2024-09-12T06:59:46.610 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.89044.log: /var/log/ceph/cluster1-client.mirror.79654.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.89044.log.gz 2024-09-12T06:59:46.610 INFO:teuthology.orchestra.run.smithi152.stderr: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.79654.log.gz 2024-09-12T06:59:46.610 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62639.log 2024-09-12T06:59:46.610 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.84860.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.84860.log.gz 2024-09-12T06:59:46.611 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87877.log 2024-09-12T06:59:46.611 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40672.log 2024-09-12T06:59:46.611 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.87877.log: /var/log/ceph/cluster2-client.mirror.62639.log: 91.9% -- replaced with /var/log/ceph/cluster2-client.mirror.62639.log.gz 2024-09-12T06:59:46.611 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86367.log 2024-09-12T06:59:46.611 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.40672.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66637.log 2024-09-12T06:59:46.611 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.86367.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.86367.log.gz 2024-09-12T06:59:46.611 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82215.log 2024-09-12T06:59:46.612 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.66637.log: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.66637.log.gz 2024-09-12T06:59:46.618 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74325.log 2024-09-12T06:59:46.620 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.82215.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.82215.log.gz 2024-09-12T06:59:46.620 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43743.log 2024-09-12T06:59:46.629 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.74325.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87046.log 2024-09-12T06:59:46.634 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.43743.log: 96.4% -- replaced with /var/log/ceph/cluster2-client.mirror.74325.log.gz 2024-09-12T06:59:46.634 INFO:teuthology.orchestra.run.smithi152.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47898.log.gz 2024-09-12T06:59:46.634 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66931.log 2024-09-12T06:59:46.634 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.87046.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82115.log 2024-09-12T06:59:46.634 INFO:teuthology.orchestra.run.smithi152.stderr: 87.4%/var/log/ceph/cluster2-client.mirror.66931.log: -- replaced with /var/log/ceph/cluster1-client.mirror.87046.log.gz 2024-09-12T06:59:46.635 INFO:teuthology.orchestra.run.smithi152.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.66931.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48433.log 2024-09-12T06:59:46.635 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-09-12T06:59:46.635 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.82115.log: gzip -5 --verbose -- /var/log/ceph/cluster2.log 2024-09-12T06:59:46.639 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.48433.log: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48433.log.gz 2024-09-12T06:59:46.639 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35800.log 2024-09-12T06:59:46.639 INFO:teuthology.orchestra.run.smithi152.stderr: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.82115.log.gz 2024-09-12T06:59:46.639 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65981.log 2024-09-12T06:59:46.639 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.admin.35800.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35800.log.gz 2024-09-12T06:59:46.639 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44876.log 2024-09-12T06:59:46.640 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.65981.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.65981.log.gz 2024-09-12T06:59:46.640 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76870.log 2024-09-12T06:59:46.640 INFO:teuthology.orchestra.run.smithi152.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.39092.log.gz/var/log/ceph/cluster1-client.mirror.44876.log: 2024-09-12T06:59:46.640 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85670.log 2024-09-12T06:59:46.646 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.76870.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.76870.log.gz 2024-09-12T06:59:46.646 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41571.log 2024-09-12T06:59:46.646 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.85670.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.85670.log.gz 2024-09-12T06:59:46.646 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83905.log 2024-09-12T06:59:46.646 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.41571.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85782.log 2024-09-12T06:59:46.646 INFO:teuthology.orchestra.run.smithi152.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.43743.log.gz 2024-09-12T06:59:46.646 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.83905.log: 87.2% -- replaced with /var/log/ceph/cluster2.log.gz 2024-09-12T06:59:46.646 INFO:teuthology.orchestra.run.smithi152.stderr: 88.3%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49823.log 2024-09-12T06:59:46.646 INFO:teuthology.orchestra.run.smithi152.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.83905.log.gz/var/log/ceph/cluster1-client.mirror.85782.log: 2024-09-12T06:59:46.646 INFO:teuthology.orchestra.run.smithi152.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.85782.log.gz 2024-09-12T06:59:46.646 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85737.log 2024-09-12T06:59:46.646 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84522.log 2024-09-12T06:59:46.647 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.85737.log: /var/log/ceph/cluster2-client.mirror.49823.log: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49823.log.gz 2024-09-12T06:59:46.648 INFO:teuthology.orchestra.run.smithi152.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.85737.log.gz 2024-09-12T06:59:46.648 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89022.log 2024-09-12T06:59:46.649 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43000.log 2024-09-12T06:59:46.669 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.84522.log: 86.9%/var/log/ceph/cluster2-client.mirror.89022.log: -- replaced with /var/log/ceph/cluster1-client.mirror.84522.log.gz 2024-09-12T06:59:46.669 INFO:teuthology.orchestra.run.smithi152.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.63823.log.gz 2024-09-12T06:59:46.669 INFO:teuthology.orchestra.run.smithi152.stderr:gzip 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.89022.log.gz 2024-09-12T06:59:46.669 INFO:teuthology.orchestra.run.smithi152.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89109.log 2024-09-12T06:59:46.669 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.43000.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60911.log 2024-09-12T06:59:46.669 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.89109.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82237.log 2024-09-12T06:59:46.669 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.60911.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.89109.log.gz 2024-09-12T06:59:46.669 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81190.log 2024-09-12T06:59:46.669 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose --/var/log/ceph/cluster2-client.mirror.82237.log: /var/log/ceph/cluster2-client.mirror.41972.log 2024-09-12T06:59:46.669 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.81190.log: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.60911.log.gz 2024-09-12T06:59:46.669 INFO:teuthology.orchestra.run.smithi152.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.81190.log.gz 2024-09-12T06:59:46.670 INFO:teuthology.orchestra.run.smithi152.stderr:gzip 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.82237.log.gz 2024-09-12T06:59:46.670 INFO:teuthology.orchestra.run.smithi152.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52217.log 2024-09-12T06:59:46.670 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82591.log 2024-09-12T06:59:46.670 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.41972.log: /var/log/ceph/cluster2-client.mirror.52217.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45036.log 2024-09-12T06:59:46.670 INFO:teuthology.orchestra.run.smithi152.stderr: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52217.log.gz 2024-09-12T06:59:46.670 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.82591.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42251.log 2024-09-12T06:59:46.670 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose --/var/log/ceph/cluster2-client.mirror.45036.log: /var/log/ceph/cluster2-client.mirror.65252.log 2024-09-12T06:59:46.670 INFO:teuthology.orchestra.run.smithi152.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.82591.log.gz 2024-09-12T06:59:46.670 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80127.log 2024-09-12T06:59:46.670 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.42251.log: /var/log/ceph/cluster2-client.mirror.65252.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.90037.log 2024-09-12T06:59:46.670 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.80127.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.65252.log.gz 2024-09-12T06:59:46.670 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83496.log 2024-09-12T06:59:46.670 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80320.log 2024-09-12T06:59:46.670 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.admin.90037.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.90037.log.gz 2024-09-12T06:59:46.671 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.83496.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44770.log 2024-09-12T06:59:46.671 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.80320.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.80320.log.gz 2024-09-12T06:59:46.671 INFO:teuthology.orchestra.run.smithi152.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.83496.log.gz 2024-09-12T06:59:46.671 INFO:teuthology.orchestra.run.smithi152.stderr: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.80127.log.gz 2024-09-12T06:59:46.671 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87443.log 2024-09-12T06:59:46.671 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.44770.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82755.log 2024-09-12T06:59:46.671 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80298.log 2024-09-12T06:59:46.671 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.82755.log: /var/log/ceph/cluster2-client.mirror.87443.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.82755.log.gz 2024-09-12T06:59:46.671 INFO:teuthology.orchestra.run.smithi152.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.87443.log.gz 2024-09-12T06:59:46.671 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83325.log 2024-09-12T06:59:46.671 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.80298.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83996.log 2024-09-12T06:59:46.671 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84725.log 2024-09-12T06:59:46.671 INFO:teuthology.orchestra.run.smithi152.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.80298.log.gz 2024-09-12T06:59:46.671 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.83996.log: /var/log/ceph/cluster2-client.mirror.83325.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.admin.39692.log.gz 2024-09-12T06:59:46.672 INFO:teuthology.orchestra.run.smithi152.stderr: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.83996.log.gz 2024-09-12T06:59:46.672 INFO:teuthology.orchestra.run.smithi152.stderr: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.83325.log.gz 2024-09-12T06:59:46.672 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73084.log 2024-09-12T06:59:46.672 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose --/var/log/ceph/cluster1-client.mirror.84725.log: /var/log/ceph/cluster1-client.mirror.84342.log 2024-09-12T06:59:46.672 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.73084.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.84725.log.gz 2024-09-12T06:59:46.672 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89366.log 2024-09-12T06:59:46.672 INFO:teuthology.orchestra.run.smithi152.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.73084.log.gz 2024-09-12T06:59:46.672 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.84342.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73141.log 2024-09-12T06:59:46.672 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.89366.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.84342.log.gz 2024-09-12T06:59:46.672 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80796.log 2024-09-12T06:59:46.672 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87332.log 2024-09-12T06:59:46.672 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.73141.log: /var/log/ceph/cluster2-client.mirror.80796.log: 91.9% -- replaced with /var/log/ceph/cluster2-client.mirror.73141.log.gz 2024-09-12T06:59:46.672 INFO:teuthology.orchestra.run.smithi152.stderr:gzip 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.80796.log.gz 2024-09-12T06:59:46.673 INFO:teuthology.orchestra.run.smithi152.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49322.log 2024-09-12T06:59:46.673 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40150.log 2024-09-12T06:59:46.673 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.87332.log: /var/log/ceph/cluster2-client.mirror.49322.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.87332.log.gz 2024-09-12T06:59:46.675 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46735.log 2024-09-12T06:59:46.675 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.40150.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74711.log 2024-09-12T06:59:46.675 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.46735.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80612.log 2024-09-12T06:59:46.675 INFO:teuthology.orchestra.run.smithi152.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.39927.log.gz 2024-09-12T06:59:46.683 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.74711.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84108.log 2024-09-12T06:59:46.696 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.80612.log: 89.3%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51629.log 2024-09-12T06:59:46.696 INFO:teuthology.orchestra.run.smithi152.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.80612.log.gz 2024-09-12T06:59:46.696 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.84108.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.84108.log.gz 2024-09-12T06:59:46.703 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53124.log 2024-09-12T06:59:46.703 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.51629.log: 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51629.log.gz 2024-09-12T06:59:46.713 INFO:teuthology.orchestra.run.smithi152.stderr: 88.9% -- replaced with /var/log/ceph/cluster1-client.admin.43616.log.gz 2024-09-12T06:59:46.714 INFO:teuthology.orchestra.run.smithi152.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.43000.log.gz 2024-09-12T06:59:46.716 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80249.log 2024-09-12T06:59:46.718 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.53124.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.40150.log.gz 2024-09-12T06:59:46.718 INFO:teuthology.orchestra.run.smithi152.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.42844.log.gz 2024-09-12T06:59:46.718 INFO:teuthology.orchestra.run.smithi152.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.45560.log.gz 2024-09-12T06:59:46.732 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84995.log 2024-09-12T06:59:46.732 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.80249.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.80249.log.gz 2024-09-12T06:59:46.743 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38592.log 2024-09-12T06:59:46.744 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.84995.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.84995.log.gz 2024-09-12T06:59:46.750 INFO:teuthology.orchestra.run.smithi152.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48235.log.gz 2024-09-12T06:59:46.750 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84680.log 2024-09-12T06:59:46.750 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.38592.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51652.log 2024-09-12T06:59:46.752 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.84680.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.84680.log.gz 2024-09-12T06:59:46.752 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38341.log 2024-09-12T06:59:46.757 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.51652.log: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.46495.log.gz 2024-09-12T06:59:46.783 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78870.log 2024-09-12T06:59:46.792 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.38341.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83376.log 2024-09-12T06:59:46.794 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.78870.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.78870.log.gz 2024-09-12T06:59:46.801 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69898.log 2024-09-12T06:59:46.802 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.83376.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.83376.log.gz 2024-09-12T06:59:46.816 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37720.log 2024-09-12T06:59:46.816 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.69898.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.69898.log.gz 2024-09-12T06:59:46.821 INFO:teuthology.orchestra.run.smithi152.stderr: 95.9% -- replaced with /var/log/ceph/cluster2-client.mirror.88165.log.gz 2024-09-12T06:59:46.822 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54535.log 2024-09-12T06:59:46.823 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.37720.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81648.log 2024-09-12T06:59:46.839 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.54535.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.38592.log.gz 2024-09-12T06:59:46.844 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64973.log 2024-09-12T06:59:46.846 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.81648.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.81648.log.gz 2024-09-12T06:59:46.847 INFO:teuthology.orchestra.run.smithi152.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.42251.log.gz 2024-09-12T06:59:46.847 INFO:teuthology.orchestra.run.smithi152.stderr: 68.6% -- replaced with /var/log/ceph/cluster2-client.mirror.37720.log.gz 2024-09-12T06:59:46.847 INFO:teuthology.orchestra.run.smithi152.stderr: 84.4% -- replaced with /var/log/ceph/cluster2-client.mirror.38068.log.gz 2024-09-12T06:59:46.857 INFO:teuthology.orchestra.run.smithi152.stderr: 93.2% -- replaced with /var/log/ceph/cluster2-client.mirror.44005.log.gz 2024-09-12T06:59:46.857 INFO:teuthology.orchestra.run.smithi152.stderr: 92.9% -- replaced with /var/log/ceph/cluster1-client.mirror.41528.log.gz 2024-09-12T06:59:46.863 INFO:teuthology.orchestra.run.smithi152.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44457.log.gz 2024-09-12T06:59:46.865 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75811.log 2024-09-12T06:59:46.873 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.64973.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.39540.log.gz 2024-09-12T06:59:46.876 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77516.log 2024-09-12T06:59:46.877 INFO:teuthology.orchestra.run.smithi152.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.74711.log.gz 2024-09-12T06:59:46.878 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.75811.log: 91.9% -- replaced with /var/log/ceph/cluster2-client.mirror.75811.log.gz 2024-09-12T06:59:46.881 INFO:teuthology.orchestra.run.smithi152.stderr: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.38637.log.gz 2024-09-12T06:59:46.884 INFO:teuthology.orchestra.run.smithi152.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.45142.log.gz 2024-09-12T06:59:46.884 INFO:teuthology.orchestra.run.smithi152.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.39029.log.gz 2024-09-12T06:59:46.891 INFO:teuthology.orchestra.run.smithi152.stderr: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.45099.log.gz 2024-09-12T06:59:46.892 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74669.log 2024-09-12T06:59:46.893 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.77516.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.77516.log.gz 2024-09-12T06:59:46.898 INFO:teuthology.orchestra.run.smithi152.stderr: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.39947.log.gz 2024-09-12T06:59:46.904 INFO:teuthology.orchestra.run.smithi152.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.87877.log.gz 2024-09-12T06:59:46.918 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50624.log 2024-09-12T06:59:46.919 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.74669.log: 91.9% -- replaced with /var/log/ceph/cluster2-client.mirror.74669.log.gz 2024-09-12T06:59:46.919 INFO:teuthology.orchestra.run.smithi152.stderr: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.53124.log.gz 2024-09-12T06:59:46.925 INFO:teuthology.orchestra.run.smithi152.stderr: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50348.log.gz 2024-09-12T06:59:46.930 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74258.log 2024-09-12T06:59:46.930 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.50624.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70125.log 2024-09-12T06:59:46.932 INFO:teuthology.orchestra.run.smithi152.stderr: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50624.log.gz 2024-09-12T06:59:46.933 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.74258.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86457.log 2024-09-12T06:59:46.933 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.70125.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.74258.log.gz 2024-09-12T06:59:46.933 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70083.log 2024-09-12T06:59:46.933 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.86457.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.86457.log.gz 2024-09-12T06:59:46.933 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68791.log 2024-09-12T06:59:46.935 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.70083.log: 92.1% -- replaced with /var/log/ceph/cluster2-client.mirror.70083.log.gz 2024-09-12T06:59:46.935 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70790.log 2024-09-12T06:59:46.937 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.68791.log: 96.0% -- replaced with /var/log/ceph/cluster2-client.mirror.70125.log.gz 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.68791.log.gz 2024-09-12T06:59:46.937 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-09-12T06:59:46.937 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65737.log 2024-09-12T06:59:46.938 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.70790.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.70790.log.gz 2024-09-12T06:59:46.938 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64839.log 2024-09-12T06:59:46.938 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86549.log 2024-09-12T06:59:46.939 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.65737.log: /var/log/ceph/cluster2-client.mirror.64839.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.64839.log.gz 2024-09-12T06:59:46.942 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.90289.log 2024-09-12T06:59:46.942 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.86549.log: 61.0% -- replaced with /var/log/ceph/cluster1-client.mirror.86549.log.gz 2024-09-12T06:59:46.942 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69169.log 2024-09-12T06:59:46.942 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.admin.90289.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.90289.log.gz 2024-09-12T06:59:46.942 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61464.log 2024-09-12T06:59:46.945 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.69169.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.69169.log.gz 2024-09-12T06:59:46.945 INFO:teuthology.orchestra.run.smithi152.stderr: 96.1% -- replaced with /var/log/ceph/cluster2-client.mirror.65737.log.gz 2024-09-12T06:59:46.945 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40381.log 2024-09-12T06:59:46.945 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose/var/log/ceph/cluster1-client.mirror.61464.log: -- /var/log/ceph/cluster1-client.admin.41144.log 2024-09-12T06:59:46.945 INFO:teuthology.orchestra.run.smithi152.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.61464.log.gz 2024-09-12T06:59:46.945 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.admin.40381.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88344.log 2024-09-12T06:59:46.952 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.admin.41144.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44657.log 2024-09-12T06:59:46.955 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.88344.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43439.log 2024-09-12T06:59:46.955 INFO:teuthology.orchestra.run.smithi152.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.88344.log.gz 2024-09-12T06:59:46.962 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.44657.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84320.log 2024-09-12T06:59:46.962 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.43439.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86636.log 2024-09-12T06:59:46.962 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.84320.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83131.log 2024-09-12T06:59:46.962 INFO:teuthology.orchestra.run.smithi152.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.84320.log.gz 2024-09-12T06:59:46.962 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.86636.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62273.log 2024-09-12T06:59:46.962 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.83131.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.86636.log.gz 2024-09-12T06:59:46.962 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62295.log 2024-09-12T06:59:46.962 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.62273.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.83131.log.gz 2024-09-12T06:59:46.962 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87399.log 2024-09-12T06:59:46.965 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.62295.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.62273.log.gz 2024-09-12T06:59:46.965 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61895.log 2024-09-12T06:59:46.965 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.87399.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.87399.log.gz 2024-09-12T06:59:46.965 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48456.log 2024-09-12T06:59:46.965 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.61895.log: 88.9% -- replaced with /var/log/ceph/cluster1-client.admin.40381.log.gz 2024-09-12T06:59:46.966 INFO:teuthology.orchestra.run.smithi152.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.61895.log.gz 2024-09-12T06:59:46.967 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81677.log 2024-09-12T06:59:46.967 INFO:teuthology.orchestra.run.smithi152.stderr: 88.5% -- replaced with /var/log/ceph/cluster1-client.admin.41144.log.gz 2024-09-12T06:59:46.980 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.48456.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40440.log 2024-09-12T06:59:46.981 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.81677.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.81677.log.gz 2024-09-12T06:59:46.984 INFO:teuthology.orchestra.run.smithi152.stderr: 95.9% -- replaced with /var/log/ceph/cluster2-client.mirror.89366.log.gz 2024-09-12T06:59:46.985 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40808.log 2024-09-12T06:59:46.986 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.40440.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39072.log 2024-09-12T06:59:46.987 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.40808.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85377.log 2024-09-12T06:59:46.988 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster2-client.mirror.39072.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86525.log 2024-09-12T06:59:46.988 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.85377.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.85377.log.gz 2024-09-12T06:59:46.990 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65715.log 2024-09-12T06:59:46.990 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.86525.log: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51652.log.gz 2024-09-12T06:59:46.990 INFO:teuthology.orchestra.run.smithi152.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.86525.log.gz 2024-09-12T06:59:46.990 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84150.log 2024-09-12T06:59:46.993 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cluster1-client.mirror.65715.log: /var/log/ceph/cluster2-client.mirror.84150.log: 91.7% -- replaced with /var/log/ceph/cluster1-client.mirror.65715.log.gz 2024-09-12T06:59:46.993 INFO:teuthology.orchestra.run.smithi152.stderr: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.84150.log.gz 2024-09-12T06:59:47.022 INFO:teuthology.orchestra.run.smithi152.stderr: 88.8% -- replaced with /var/log/ceph/cluster1-client.mirror.44302.log.gz 2024-09-12T06:59:47.022 INFO:teuthology.orchestra.run.smithi152.stderr: 93.1% -- replaced with /var/log/ceph/cluster1-client.mirror.41571.log.gz 2024-09-12T06:59:47.033 INFO:teuthology.orchestra.run.smithi152.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.64973.log.gz 2024-09-12T06:59:47.040 INFO:teuthology.orchestra.run.smithi152.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.62295.log.gz 2024-09-12T06:59:47.041 INFO:teuthology.orchestra.run.smithi152.stderr: 93.2% -- replaced with /var/log/ceph/cluster2-client.mirror.45036.log.gz 2024-09-12T06:59:47.041 INFO:teuthology.orchestra.run.smithi152.stderr: 88.6% -- replaced with /var/log/ceph/cluster1-client.mirror.43439.log.gz 2024-09-12T06:59:47.044 INFO:teuthology.orchestra.run.smithi152.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49322.log.gz 2024-09-12T06:59:47.087 INFO:teuthology.orchestra.run.smithi152.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48456.log.gz 2024-09-12T06:59:47.100 INFO:teuthology.orchestra.run.smithi152.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.54535.log.gz 2024-09-12T06:59:47.137 INFO:teuthology.orchestra.run.smithi152.stderr: 92.9% -- replaced with /var/log/ceph/cluster2-client.mirror.44770.log.gz 2024-09-12T06:59:47.138 INFO:teuthology.orchestra.run.smithi152.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.86962.log.gz 2024-09-12T06:59:47.254 INFO:teuthology.orchestra.run.smithi152.stderr: 19.3% -- replaced with /var/log/ceph/cluster1-client.mirror.1.37902.log.gz 2024-09-12T06:59:47.254 INFO:teuthology.orchestra.run.smithi152.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.39792.log.gz 2024-09-12T06:59:47.260 INFO:teuthology.orchestra.run.smithi152.stderr: 93.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44876.log.gz 2024-09-12T06:59:47.330 INFO:teuthology.orchestra.run.smithi152.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.45720.log.gz 2024-09-12T06:59:47.397 INFO:teuthology.orchestra.run.smithi152.stderr: 96.7% -- replaced with /var/log/ceph/cluster2-client.mirror.39582.log.gz 2024-09-12T06:59:47.417 INFO:teuthology.orchestra.run.smithi152.stderr: 92.9% -- replaced with /var/log/ceph/cluster1-client.mirror.44923.log.gz 2024-09-12T06:59:47.419 INFO:teuthology.orchestra.run.smithi152.stderr: 92.9% -- replaced with /var/log/ceph/cluster1-client.mirror.40988.log.gz 2024-09-12T06:59:47.449 INFO:teuthology.orchestra.run.smithi152.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.39072.log.gz 2024-09-12T06:59:47.531 INFO:teuthology.orchestra.run.smithi152.stderr: 94.8% -- replaced with /var/log/ceph/cluster2-client.mirror.41593.log.gz 2024-09-12T06:59:47.657 INFO:teuthology.orchestra.run.smithi152.stderr: 92.9% -- replaced with /var/log/ceph/cluster1-client.mirror.40672.log.gz 2024-09-12T06:59:47.802 INFO:teuthology.orchestra.run.smithi152.stderr: 92.6% -- replaced with /var/log/ceph/cluster1-client.mirror.44657.log.gz 2024-09-12T06:59:47.805 INFO:teuthology.orchestra.run.smithi152.stderr: 96.9% -- replaced with /var/log/ceph/cluster2-client.mirror.43553.log.gz 2024-09-12T06:59:47.866 INFO:teuthology.orchestra.run.smithi096.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-mon.a.log.gz 2024-09-12T06:59:47.913 INFO:teuthology.orchestra.run.smithi152.stderr: 88.6% -- replaced with /var/log/ceph/cluster1-client.admin.41686.log.gz 2024-09-12T06:59:47.914 INFO:teuthology.orchestra.run.smithi152.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.41972.log.gz 2024-09-12T06:59:48.466 INFO:teuthology.orchestra.run.smithi152.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.39904.log.gz 2024-09-12T06:59:49.156 INFO:teuthology.orchestra.run.smithi152.stderr: 92.8% -- replaced with /var/log/ceph/cluster1-client.mirror.44833.log.gz 2024-09-12T06:59:49.164 INFO:teuthology.orchestra.run.smithi152.stderr: 92.6% -- replaced with /var/log/ceph/cluster1-client.mirror.44700.log.gz 2024-09-12T06:59:49.785 INFO:teuthology.orchestra.run.smithi152.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.0.38002.log.gz 2024-09-12T06:59:56.666 INFO:teuthology.orchestra.run.smithi152.stderr: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43675.log.gz 2024-09-12T06:59:57.562 INFO:teuthology.orchestra.run.smithi152.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.40808.log.gz 2024-09-12T06:59:59.410 INFO:teuthology.orchestra.run.smithi152.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.39720.log.gz 2024-09-12T07:00:00.062 INFO:teuthology.orchestra.run.smithi152.stderr: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46518.log.gz 2024-09-12T07:00:00.850 INFO:teuthology.orchestra.run.smithi152.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.40354.log.gz 2024-09-12T07:00:01.140 INFO:teuthology.orchestra.run.smithi152.stderr: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.40082.log.gz 2024-09-12T07:00:02.241 INFO:teuthology.orchestra.run.smithi152.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46735.log.gz 2024-09-12T07:00:03.223 INFO:teuthology.orchestra.run.smithi152.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.39992.log.gz 2024-09-12T07:00:03.585 INFO:teuthology.orchestra.run.smithi152.stderr: 95.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46830.log.gz 2024-09-12T07:00:03.845 INFO:teuthology.orchestra.run.smithi152.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.41080.log.gz 2024-09-12T07:00:04.071 INFO:teuthology.orchestra.run.smithi152.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.40717.log.gz 2024-09-12T07:00:04.640 INFO:teuthology.orchestra.run.smithi152.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.38447.log.gz 2024-09-12T07:00:05.599 INFO:teuthology.orchestra.run.smithi152.stderr: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.41172.log.gz 2024-09-12T07:00:07.146 INFO:teuthology.orchestra.run.smithi152.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.45997.log.gz 2024-09-12T07:00:07.555 INFO:teuthology.orchestra.run.smithi152.stderr: 96.1% -- replaced with /var/log/ceph/cluster2-client.mirror.38682.log.gz 2024-09-12T07:00:07.957 INFO:teuthology.orchestra.run.smithi152.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46930.log.gz 2024-09-12T07:00:08.115 INFO:teuthology.orchestra.run.smithi152.stderr: 96.1% -- replaced with /var/log/ceph/cluster2-client.mirror.46292.log.gz 2024-09-12T07:00:08.665 INFO:teuthology.orchestra.run.smithi152.stderr: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45867.log.gz 2024-09-12T07:00:08.855 INFO:teuthology.orchestra.run.smithi152.stderr: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.40440.log.gz 2024-09-12T07:00:08.905 INFO:teuthology.orchestra.run.smithi152.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46076.log.gz 2024-09-12T07:00:10.112 INFO:teuthology.orchestra.run.smithi152.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.38341.log.gz 2024-09-12T07:00:10.402 INFO:teuthology.orchestra.run.smithi152.stderr: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.41714.log.gz 2024-09-12T07:00:11.271 INFO:teuthology.orchestra.run.smithi152.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.39137.log.gz 2024-09-12T07:00:11.553 INFO:teuthology.orchestra.run.smithi152.stderr: 90.5% -- replaced with /var/log/ceph/cluster2-mon.a.log.gz 2024-09-12T07:00:12.445 INFO:teuthology.orchestra.run.smithi152.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46605.log.gz 2024-09-12T07:00:12.475 INFO:teuthology.orchestra.run.smithi152.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.45801.log.gz 2024-09-12T07:00:12.572 INFO:teuthology.orchestra.run.smithi152.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.45925.log.gz 2024-09-12T07:00:12.912 INFO:teuthology.orchestra.run.smithi152.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.41616.log.gz 2024-09-12T07:00:13.883 INFO:teuthology.orchestra.run.smithi152.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46169.log.gz 2024-09-12T07:00:13.907 INFO:teuthology.orchestra.run.smithi152.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.43576.log.gz 2024-09-12T07:00:14.113 INFO:teuthology.orchestra.run.smithi152.stderr: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.39224.log.gz 2024-09-12T07:00:15.759 INFO:teuthology.orchestra.run.smithi152.stderr: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.38772.log.gz 2024-09-12T07:00:15.935 INFO:teuthology.orchestra.run.smithi152.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.46370.log.gz 2024-09-12T07:00:18.582 INFO:teuthology.orchestra.run.smithi152.stderr: 94.6% -- replaced with /var/log/ceph/cluster2-client.mirror.37941.log.gz 2024-09-12T07:00:19.702 INFO:teuthology.orchestra.run.smithi152.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.39605.log.gz 2024-09-12T07:00:52.744 INFO:teuthology.orchestra.run.smithi152.stderr: 94.8% -- replaced with /var/log/ceph/cluster2-client.mirror.37897.log.gz 2024-09-12T07:01:02.790 INFO:teuthology.orchestra.run.smithi152.stderr: 94.2% -- replaced with /var/log/ceph/cluster1-client.mirror.1.37941.log.gz 2024-09-12T07:01:24.245 INFO:teuthology.orchestra.run.smithi096.stderr: 94.0% -- replaced with /var/log/ceph/cluster1-osd.1.log.gz 2024-09-12T07:01:27.618 INFO:teuthology.orchestra.run.smithi096.stderr: 94.1% -- replaced with /var/log/ceph/cluster1-osd.2.log.gz 2024-09-12T07:01:41.788 INFO:teuthology.orchestra.run.smithi096.stderr: 94.0% -- replaced with /var/log/ceph/cluster1-osd.0.log.gz 2024-09-12T07:01:41.791 INFO:teuthology.orchestra.run.smithi096.stderr: 2024-09-12T07:01:41.791 INFO:teuthology.orchestra.run.smithi096.stderr:real 1m57.923s 2024-09-12T07:01:41.791 INFO:teuthology.orchestra.run.smithi096.stderr:user 3m3.821s 2024-09-12T07:01:41.791 INFO:teuthology.orchestra.run.smithi096.stderr:sys 0m7.515s 2024-09-12T07:02:17.273 INFO:teuthology.orchestra.run.smithi152.stderr: 94.3% -- replaced with /var/log/ceph/cluster1-client.mirror.0.37897.log.gz 2024-09-12T07:09:49.553 INFO:teuthology.orchestra.run.smithi152.stderr: 94.3% -- replaced with /var/log/ceph/cluster2-osd.1.log.gz 2024-09-12T07:10:05.629 INFO:teuthology.orchestra.run.smithi152.stderr: 94.3% -- replaced with /var/log/ceph/cluster2-osd.2.log.gz 2024-09-12T07:10:33.274 INFO:teuthology.orchestra.run.smithi152.stderr: 94.2% -- replaced with /var/log/ceph/cluster2-osd.0.log.gz 2024-09-12T07:10:33.276 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-09-12T07:10:33.276 INFO:teuthology.orchestra.run.smithi152.stderr:real 10m49.377s 2024-09-12T07:10:33.276 INFO:teuthology.orchestra.run.smithi152.stderr:user 5m4.876s 2024-09-12T07:10:33.276 INFO:teuthology.orchestra.run.smithi152.stderr:sys 0m17.206s 2024-09-12T07:10:33.277 INFO:tasks.ceph:Archiving logs... 2024-09-12T07:10:33.278 DEBUG:teuthology.misc:Transferring archived files from smithi096:/var/log/ceph to /home/teuthworker/archive/dis-2024-09-12_04:46:10-rbd-wip-dis-testing-distro-default-smithi/7902364/remote/smithi096/log 2024-09-12T07:10:33.279 DEBUG:teuthology.orchestra.run.smithi096:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-12T07:10:59.841 DEBUG:teuthology.misc:Transferring archived files from smithi152:/var/log/ceph to /home/teuthworker/archive/dis-2024-09-12_04:46:10-rbd-wip-dis-testing-distro-default-smithi/7902364/remote/smithi152/log 2024-09-12T07:10:59.842 DEBUG:teuthology.orchestra.run.smithi152:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-12T07:11:42.402 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-12T07:11:42.413 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-12T07:11:42.413 DEBUG:teuthology.orchestra.run.smithi096:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-12T07:11:42.416 DEBUG:teuthology.orchestra.run.smithi152:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-12T07:11:42.619 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-12T07:11:42.628 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-12T07:11:42.628 DEBUG:teuthology.orchestra.run.smithi096:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-12T07:11:42.631 DEBUG:teuthology.orchestra.run.smithi152:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-12T07:11:42.645 INFO:teuthology.orchestra.run.smithi096.stderr:bash: line 1: ntpq: command not found 2024-09-12T07:11:42.649 INFO:teuthology.orchestra.run.smithi152.stderr:bash: line 1: ntpq: command not found 2024-09-12T07:11:42.677 INFO:teuthology.orchestra.run.smithi152.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-12T07:11:42.677 INFO:teuthology.orchestra.run.smithi152.stdout:=============================================================================== 2024-09-12T07:11:42.677 INFO:teuthology.orchestra.run.smithi152.stdout:^* hv01.front.sepia.ceph.com 3 6 377 62 +668us[ +668us] +/- 44ms 2024-09-12T07:11:42.677 INFO:teuthology.orchestra.run.smithi152.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 63 -112us[ -113us] +/- 72ms 2024-09-12T07:11:42.677 INFO:teuthology.orchestra.run.smithi152.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 61 -1049us[-1049us] +/- 80ms 2024-09-12T07:11:42.677 INFO:teuthology.orchestra.run.smithi152.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-12T07:11:42.708 INFO:teuthology.orchestra.run.smithi096.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-12T07:11:42.708 INFO:teuthology.orchestra.run.smithi096.stdout:=============================================================================== 2024-09-12T07:11:42.708 INFO:teuthology.orchestra.run.smithi096.stdout:^* hv01.front.sepia.ceph.com 3 6 377 64 +639us[ +657us] +/- 44ms 2024-09-12T07:11:42.709 INFO:teuthology.orchestra.run.smithi096.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 2 -188us[ -188us] +/- 73ms 2024-09-12T07:11:42.709 INFO:teuthology.orchestra.run.smithi096.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 4 -1028us[-1028us] +/- 81ms 2024-09-12T07:11:42.709 INFO:teuthology.orchestra.run.smithi096.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-12T07:11:42.710 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-12T07:11:42.719 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-12T07:11:42.720 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-12T07:11:42.728 DEBUG:teuthology.orchestra.run.smithi096:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-12T07:11:42.842 DEBUG:teuthology.orchestra.run.smithi152:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-12T07:11:42.924 DEBUG:teuthology.orchestra.run.smithi096:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-12T07:11:42.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-12T07:11:42.955 DEBUG:teuthology.orchestra.run.smithi152:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-12T07:11:42.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-12T07:11:42.985 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-12T07:11:42.994 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-12T07:11:43.033 INFO:teuthology.task.internal:Duration was 3078.257374 seconds 2024-09-12T07:11:43.034 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-12T07:11:43.054 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-12T07:11:43.054 DEBUG:teuthology.orchestra.run.smithi096:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-12T07:11:43.057 DEBUG:teuthology.orchestra.run.smithi152:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-12T07:11:43.121 INFO:teuthology.orchestra.run.smithi096.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-12T07:11:43.145 INFO:teuthology.orchestra.run.smithi152.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-12T07:11:43.529 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-12T07:11:43.530 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi096.front.sepia.ceph.com 2024-09-12T07:11:43.530 DEBUG:teuthology.orchestra.run.smithi096:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-12T07:11:43.551 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi152.front.sepia.ceph.com 2024-09-12T07:11:43.552 DEBUG:teuthology.orchestra.run.smithi152:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-12T07:11:43.574 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-12T07:11:43.574 DEBUG:teuthology.orchestra.run.smithi096:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-12T07:11:43.594 DEBUG:teuthology.orchestra.run.smithi152:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-12T07:11:43.642 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-12T07:11:43.643 DEBUG:teuthology.orchestra.run.smithi096:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-12T07:11:43.662 DEBUG:teuthology.orchestra.run.smithi152:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-12T07:11:43.806 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-12T07:11:43.815 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-12T07:11:43.816 DEBUG:teuthology.orchestra.run.smithi096:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-12T07:11:43.840 DEBUG:teuthology.orchestra.run.smithi152:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-12T07:11:43.894 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-12T07:11:43.903 DEBUG:teuthology.orchestra.run.smithi096:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-12T07:11:43.906 DEBUG:teuthology.orchestra.run.smithi152:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-12T07:11:43.938 INFO:teuthology.orchestra.run.smithi096.stdout:kernel.core_pattern = core 2024-09-12T07:11:43.941 INFO:teuthology.orchestra.run.smithi152.stdout:kernel.core_pattern = core 2024-09-12T07:11:43.962 DEBUG:teuthology.orchestra.run.smithi096:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-12T07:11:44.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-12T07:11:44.017 DEBUG:teuthology.orchestra.run.smithi152:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-12T07:11:44.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-12T07:11:44.032 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-12T07:11:44.042 INFO:teuthology.task.internal:Transferring archived files... 2024-09-12T07:11:44.043 DEBUG:teuthology.misc:Transferring archived files from smithi096:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/dis-2024-09-12_04:46:10-rbd-wip-dis-testing-distro-default-smithi/7902364/remote/smithi096 2024-09-12T07:11:44.043 DEBUG:teuthology.orchestra.run.smithi096:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-12T07:11:44.091 DEBUG:teuthology.misc:Transferring archived files from smithi152:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/dis-2024-09-12_04:46:10-rbd-wip-dis-testing-distro-default-smithi/7902364/remote/smithi152 2024-09-12T07:11:44.091 DEBUG:teuthology.orchestra.run.smithi152:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-12T07:11:44.125 INFO:teuthology.task.internal:Removing archive directory... 2024-09-12T07:11:44.125 DEBUG:teuthology.orchestra.run.smithi096:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-12T07:11:44.128 DEBUG:teuthology.orchestra.run.smithi152:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-12T07:11:44.176 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-12T07:11:44.188 INFO:teuthology.task.internal:Not uploading archives. 2024-09-12T07:11:44.188 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-12T07:11:44.197 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-12T07:11:44.197 DEBUG:teuthology.orchestra.run.smithi096:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-12T07:11:44.200 DEBUG:teuthology.orchestra.run.smithi152:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-12T07:11:44.212 INFO:teuthology.orchestra.run.smithi096.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 12 07:11 /home/ubuntu/cephtest 2024-09-12T07:11:44.231 INFO:teuthology.orchestra.run.smithi152.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 12 07:11 /home/ubuntu/cephtest 2024-09-12T07:11:44.232 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-12T07:11:44.242 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-12T07:11:44.264 INFO:teuthology.run:Summary data: description: rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{centos_latest} workloads/rbd-mirror-stress-workunit-min-compat-client-octopus} duration: 3078.2573742866516 flavor: default owner: scheduled_dis@teuthology success: true 2024-09-12T07:11:44.264 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-12T07:11:44.364 INFO:teuthology.run:pass